File Compression and Archiving

The following tips deal with the use of tar, gzip, bzip2 and zip

make a tarball of certain types of files
You can make a tarball of only certain types of files from a directory with the following one-liner:

find reports/ -name "*.txt" | xargs tar -zcpf reports.tar.gz

Change the compression type to bzip2 and you'll have a smaller file:

find reports/ -name "*.txt" | xargs tar -jcpf reports.tar.bz2

untar in a different directory
If you've got a gzipped tarball and you want to untar it in a directory other than the one you're in, do the following:

zcat file.tar.gz | ( cd ./otherdir; tar xvf - )

extract individual files from a tarball
If you need a file that you've put into a tarball and you don't want to extract the whole file, you can do the following.
First, get a list of the files and find the one you want

tar -zltf file.tar.gz

Then extract the one you want

tar zxvf file.tar.gz indiv.file

backup everything with tar
To make a backup of everything in a particular directory, first do this

ls -a > backup.all

If you don't really want *everything*, you can also edit backup.all and get rid of things you don't want
To make the tarball, just do this:

tar -cvf newtarfile.tar `cat filelist`

(remember, those are backtics)


incremental backups with tar
An incremental backup is one that includes the new files you've created each day. First you need create a tarball with everything, as we did in the previous example, or just by creating a tarball of the entire directory.

tar cvzf homedir-complete.tar.gz /home/homedir/

Then, you can run the following as a cron job every day. It will add the new files to the homedir-complete.tar.gz tarball

find /home/homedir -mtime -1 -print | 
tar cvzf homedir_complete.tar.gz -T -

(don't forget that dash '-' at the end!)


zip syntax
Most Linux distributions come with zip, the most popular file compression format in the MS Windows world. The command line arguments are not the same as when you're creating a tar.gz, however. To create a zipped file on Linux, do this:

zip -b ./ file.zip *.png

This would create a zipped file of all the *.pngs in that particular directory. Keep in mind:

• "./" means the directory you're in

• As you can see, you can use wildcards and extension names

tar and untar over ssh
You can use tar combined with ssh to move files more easily over a local network or even the Internet. In the following example, we will take our 'development' web work and transfer them to the production server.

tar cf - *.png | ssh -l username production_server 
"( cd /home/website/html/images/ ; tar xf - ) "

The process can also be reversed to get remote files here.

ssh -l username production_server
"( cd /path/to/files; tar cf - *.png ) " | tar xf

Microsoft Windows Server 2012 : Hyper-V missing features and potential fixes

For all the new features and enhancements to Windows Server 2012 Hyper-V, the virtualization platform still has its shortcomings.

Having worked with Hyper-V since the early betas, I appreciate the speed at which Microsoft adds new features. With each Hyper-V release, however, I still see a few areas of improvement that could add key functionality and ease administration. Below, I look at three of these shortcomings in Windows Server 2012 Hyper-V and offer some practical fixes.

1. Poor Quick Migration between hosts with different processor architectures

It’s possible to migrate virtual machines (VMs) between Windows Server 2012 Hyper-V hosts with different processor types, as long as they are within the same family (e.g., Intel to Intel or AMD to AMD). But it’s not possible to move VMs between processor vendors without first shutting down the VM and moving it to the alternate host. The problem is, if a VM has a large Virtual Hard Disk, the amount of downtime can be lengthy, as it’s moved to the new Windows Server 2012 Hyper-V host.

How should it work?
For Windows workloads, I understand there needs to be some recognition of the new processor architecture, so a reboot or two may be necessary. But the method below would potentially cut the downtime from hours to minutes.

1. I foresee Microsoft using the same method of Live Migration between Windows Server 2012 Hyper-V hosts or a similar process to that of Quick Storage Migration in Windows Server 2008 R2 SP1, which takes a Hyper-V VSS Writer snapshot of the VM and moves the data to the alternate host, while the source VM is still running.
2. Once the data is moved, the new VM restarts offline, allowing for the VM to recognize the processor architecture, then shuts down.
3. Next, the source VM shuts down, and a final Hyper-V VSS Writer snapshot is taken. Any remaining changes are transitioned over, and the VM on the destination host is restarted and brought online.

The process is very similar to a physical-to-virtual migration, where you keep an Intel or AMD architecture host online and migrate it to a virtual host with a different processor.

Benefits of this method
This process would be good for migrating large VMs between hosts, because large VHD/VHDX files can take a long time to move. It would also benefit low-bandwidth environments, because the source VM would stay online during the migration.

2. Inadequate Live/Quick Migration of VMs between different Hyper-V versions

Hyper-V’s rapidly evolving features have kept up with the needs of modern application resource requirements. But this fast pace also creates a constant need to migrate hosts and VMs to the latest Hyper-V version.

In most cases, this process requires a shutdown of the VMs. Then, you must migrate them offline to the updated host and install the new Integration Components.

How should it work?
Similarly to the above scenario, migrating VMs between Hyper-V versions probably necessitates a reboot to install new Integration Components. But extended downtime should not be part of the process. Below is how I see it working in the future.

1. Hyper-V performs a Hyper-V VSS Writer-assisted snapshot and moves the data from the source VM, while it is still running, until all data is moved to the new host;
2. It then automatically starts the new destination VM offline (the virtual NIC is not connected) and installs Integration Components; then
3. Shuts down the source VM once the first pass of data is moved to the destination host;
4. Moves the remaining changes from the source VM to the destination VM, so both virtual machines are synchronized; and finally
5. Restarts the destination VM.

Benefits of this method
The source VM remains online until all data is moved to the new host. At the same time, this method reduces downtime and any strain that causes.

3. Imperfect hot-add memory allocation with a running VM

It should come to no surprise that, over the years, server workloads have consumed more resources, including disk space, CPU and RAM. Microsoft added a hot-add disk capability to Hyper-V in Windows Server 2008 R2, and hot-add memory is now supported in Windows Server 2012 Hyper-V, as long as the VM utilizes Dynamic Memory. But it is still not possible to hot-add memory if the VM uses a statically assigned or fixed-memory allocation.

By only enabling hot-add memory for VMs that utilize Dynamic Memory, Microsoft misses workloads that would most likely need an uninterrupted memory increase such as databases or even Java-based applications. The latter are not recommended for use with or supported by Dynamic Memory, because they use memory resources fully and cannot call for more memory from the dynamic range set for the VM.

Workloads that are memory hungry are usually configured with a fixed amount, and they are just the ones that could use more memory on the fly, without interruption. For those reasons, I still think the addition of hot-add memory is incomplete and only partly reduces the administrative burden for IT pros.

How should it work?
Using the capabilities already in Windows Server, Microsoft should include the ability to adjust the memory maximums for fixed-memory allocations.

The results of this fix
This capability would eliminate another manual process that incurs downtime. It would also allow for more automated remediation processes (with the use of System Center Operations Manager), which would accommodate the periodic need for memory-hungry workloads to receive additional resources. Finally, it would meet the needs of workloads with unpredictable memory requirements, such as databases, that are the most sensitive to downtime.

Don’t get me wrong: Windows Server 2012 Hyper-V includes some excellent additions. But, with every release, there are bound to be a few features that miss the cut. The options above would dramatically reduce migration and administrative time, but they are unfortunately not available…yet.

What is TCP-MSS in Junos

Tcp-mss is negotiated as part of the TCP 3-way handshake. It limits the maximum size of a TCP segment to better fit the maximum transmission unit (MTU) limits on a network.

This is especially important for VPN traffic as the IPsec encapsulation overhead, along with the IP and frame overhead, can cause the resulting Encapsulating Security Payload (ESP) packet to exceed the MTU of the physical interface, thus causing fragmentation. Fragmentation increases bandwidth and device resources and is always best avoided.

Note that the value of 1350 is a recommended starting point for most Ethernet-based networks with an MTU of 1500 or greater. This value may need to be altered if any device in the path has lower MTU, or if there is any added overhead such as Point-to-Point Protocol (PPP), Frame Relay, etc.

As a general rule, you may need to experiment with different tcp-mss values to obtain optimal performance.

vCenter and vCloud Management Design - Management Separation

While at a customer site this past week, I was confronted with a situation. But before I get to that, lets talk about vCenter and vCloud Design. First thing is first, you should be vaguely familiar with vCloud Architecture Toolkit (vCAT). One important topic it discusses is the placement and use of vCenter when it comes to vCloud Director. It's a recommended practice to have 2 vCenter servers in a vCloud environment. Use 1 vCenter server for hosting Datacenters/Clusters/VMs that are relevant to vSphere and vCloud Infrastructure Components. Use another vCenter server for hosting vCloud Resources. Why's this? Separation of management domains. It's important to know that vSphere and vCloud are different animals. Just because you are a vSphere admin, it doesn't make you a vCloud admin. By separating the two environments, you are letting vSphere admins access VMs that are outside the Cloud, and manage VMs that are considered vCloud Infrastructure. vCenter becomes abstracted. ESXi abstracts the hardware layer, and vCenter is the central management point. vCloud Director abstracts the resources that belong to vCenter and present those to vCloud as Provider Virtual Datacenters. Saves vSphere Admins from themselves. Have you've ever watched what happens when you add a vCenter server to vCloud Director? vCloud Director takes charge. It does it's own thing by creating folders, resource pools, port groups, appliances, etc. Everything that is created by vCloud has a set of characters that proceed it to become unique identifiers. If a vSphere admin has access to a Distributed Virtual Switch, and notices some random portgroup ending with HFE2342-FEF2123NJE-234, he is probably tempted to delete it. If a user goes crazy and starts deleting objects directly from vCenter without vCloud's knowledge, its havoc. Relieve Stress on vCenter. As Duncan pointed out below in the comments, if a tenant of the cloud is issuing a bunch of requests, it could possible render the vCenter server unusable. By separating out the workload among 2 vCenter functions, you will not impact a vCenter server responsible for management functions. So lets take a look at 2 vCenter vCloud environment where there is already a management cluster for other applications. This management cluster already has a vCenter Server, SQL Server, vCOPS, AD/DNS, etc. and probably manages other cluster in the datacenter. If the resources are available, you will want to create a cluster called a vCloud Management Cluster. This Management cluster will house the second vCenter Server, SQL, vShield Manager, vCD Cells. There is a second SQL server because the vCloud vCenter, vCloud vCenter Update Manager, and vCloud Director applications will all need access to a database. It's best to have a second because using a single SQL server in a different cluster can cause latency int he applications or unexpected downtime. As depicted in the diagram, the Management vCenter Server owns the Management Cluster and the vCloud Management Cluster. The vCloud vCenter Server own the vCloud Resource clusters. This is how it would look in vCenter: Here's another instance where we see a 2 vCenter vCloud environment. Instead of having a dedicated vCloud Management Cluster, it's integrated into a different management cluster. This management cluster will need to have ample space to satisfy HA requirements. As we can see in the logical diagram, all the VMs before have been listed (I didn't show AD/DNS, but that should be in there as well) except there is an orange and red box around the second SQL Server. This depiction means that a 2nd server may or may not be necessary depending on your requirements. Here is a resemblance in vCenter: So back to my dilemma this past week. I'm standing up a vCloud POC environment for a customer and they only purchased 1 vCenter license. What's the best approach? Well of course the single vCenter and SQL server are going to manage all clusters. But how do we get around some things we mentioned before about separation of management domains, and saving a vSphere admin from himself? Here is the simple logical design, but skip on down to the vCenter example. So you might be looking at this diagram and saying, "yeah, so what?". There was thought into what you see below. Since we only have a single vCenter to manage all the clusters, we need to create a separation of management domains. This is done by creating a second logical datacenter and putting the vCD resource clusters in that datacenter. Using role based access control (RBAC) we can allow certain AD users/groups to only access the datacenter that is relevant to them. Therefore, an AD group such as "Cloud Admins" can access the vCloud-Resources Datacenter object when logging into vCenter. At the same time, creating this second logical datacenter is saving a vSphere admin from themselves. If this were all pooled into a single logical datacenter, a vSphere admin would be looking at a mess of multiple virtual distributed switches and folders. There would be a vDS assigned to the management cluster, and a vDS or two assigned to vCloud Resources, and perhaps a recovery vSwitch somewhere. All of these would be visible from a single view (BAD!). Since a virtual distributed switch is tied to logical datacenter, we get a clean separation. When a vSphere admin looks at the Networking tab under the vCloud-Resources datacenter, they will only see things relevant to vCloud and vice-versa. This logical datacenter separation allows you to safely use 1 vCenter server in your vCloud environment.