Thursday, December 16, 2010

lsblk(8)

lsblk(8) is a new util that will be available in util-linux-2.19 (coming soon;-). The util lists all block devices as a tree. This output is very useful on machines with complicated storage setup (e.g. RAID, dm-crypt, ...). It's so useful that we'll probably backport lsblk(8) to RHEL-6 to make life easier for people who need to debug their systems.

The original idea comes from "dmsetup ls --tree", but lsblk(8) is better :-) It uses "holders" and "slaves" from /sys filesystem. This means that the util is usable without root permissions and it works for all types of block devices (dmsetup uses DM ioctls).

For example my laptop with dm-crypted $HOME and partitioned RAID0 (md8) on scsi_debug device (sdc):
$ lsblk
NAME MAJ:MIN RA SIZE RO MOUNTPOINT
sda 8:0 0 93.2G 0
├─sda1 8:1 0 102M 0 /mnt/test
├─sda2 8:2 0 1K 0
├─sda3 8:3 0 2.3G 0 [SWAP]
├─sda4 8:4 0 76.2G 0 /
├─sda5 8:5 0 10G 0
│ └─kzak-home (dm-0) 253:0 0 10G 0 /home/kzak
└─sda6 8:6 0 4.7G 0 /boot
sdc 8:32 0 500M 0
├─sdc1 8:33 0 250M 0
│ └─md8 9:8 0 498.9M 0
│ ├─md8p1 259:0 0 100M 0
│ ├─md8p2 259:1 0 100M 0
│ └─md8p3 259:2 0 297.9M 0
└─sdc2 8:34 0 249M 0
└─md8 9:8 0 498.9M 0
├─md8p1 259:0 0 100M 0
├─md8p2 259:1 0 100M 0
└─md8p3 259:2 0 297.9M 0
You can also list more details about devices. The next example is from scsi_debug device with 4KiB sectors and enabled alignment offset:
$ lsblk --topology /dev/sdb
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED
sdb 3584 4096 32768 4096 512 0 cfq
├─sdb1 0 4096 32768 4096 512 0 cfq
└─sdb2 0 4096 32768 4096 512 0 cfq
lsblk(8) is also usable in script, for example:
$ lsblk --nodeps --noheading --raw -o ALIGNMENT /dev/sdb
3584
returns only alignment-offset for device sdb.

Thanks to Milan Broz (from Red Hat) who is author of the original lsblk(8) prototype.

Thursday, December 2, 2010

util-linux is without -ng

The util-linux-ng project was officially merged into util-linux in last days. It means that there is no more -ng fork. Fortunately, the change was pretty simple, because the original util-linux project was inactive in last four years. It was enough to rename mailing list, git repository and directories at kernel.org. Note that the old addresses still work and all is redirected to the new stuff.

The last remaining problem is to rename the mailing list at gmane.org (http://news.gmane.org/gmane.linux.utilities.util-linux-ng/). I don't want to lost the list history, so the renaming seems as a better way than remove old list and add a new list (without -ng). Let's hope that gmane admins will be able to do the change.

See
http://article.gmane.org/gmane.linux.file-systems/49137
for more details about new URLs and addresses.

Thanks to Kay, Adrian, John and David.

Thursday, July 15, 2010

findmnt(8)

I released util-linux-ng 2.18 two weeks ago. There is many changes, for example completely new libmount (not stable API yet), new fsfreeze(8) and findmnt(8) utils and some important changes in fdisk(8).

From my point of view the most attractive for end users is findmnt(8). This new util is a command line interface to the libmount library, the util is able to search in /etc/fstab, /etc/mtab or /proc/self/mountinfo.

Default output (mounted filesystems):
$ findmnt
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda4 ext3 rw,noatime,errors=co
├─/proc /proc proc rw,relatime
│ ├─/proc/bus/usb /proc/bus/usb usbfs rw,relatime
│ ├─/proc/sys/fs/binfmt_misc none binfmt_m rw,relatime
│ └─/proc/fs/nfsd nfsd nfsd rw,relatime
├─/sys /sys sysfs rw,relatime
├─/dev udev devtmpfs rw,relatime,size=197
│ ├─/dev/pts devpts devpts rw,relatime,gid=5,mo
│ └─/dev/shm tmpfs tmpfs rw,relatime
├─/boot /dev/sda1 ext3 rw,noatime,errors=co
├─/home/kzak /dev/mapper/kzak-home ext4 rw,noatime,barrier=1
│ └─/home/kzak/.gvfs gvfs-fuse-daemon fuse.gvf rw,nosuid,nodev,rela
├─/var/lib/nfs/rpc_pipefs sunrpc rpc_pipe rw,relatime
├─/mnt/foo //sr.net.home/foo cifs rw,relatime,mand,unc
└─/mnt/test /dev/sda6 btrfs rw,relatime
Get info about a mountpoint:
$ findmnt /home/kzak
TARGET SOURCE FSTYPE OPTIONS
/home/kzak /dev/mapper/kzak-home ext4 rw,noatime,barrier=1,data=ordered
Get all mounted extN filesystems:
$ findmnt -t ext4,ext3
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda4 ext3 rw,noatime,errors=continue,user_xattr
/boot /dev/sda1 ext3 rw,noatime,errors=continue,user_xattr
/home/kzak /dev/mapper/kzak-home ext4 rw,noatime,barrier=1,data=ordered
The same thing, but from fstab:
$ findmnt --fstab -t ext4,ext3
TARGET SOURCE FSTYPE OPTIONS
/ UUID=d3a8f783-df75-4dc8-9163-975a891052c0 ext3 noatime,defaults
/boot UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe ext3 noatime,defaults
/home/kzak /dev/mapper/kzak-home ext4 noatime,defaults
Don't like LABLEs/UUIDs?
$ findmnt --fstab --evaluate -t ext4,ext3
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda4 ext3 noatime,defaults
/boot /dev/sda1 ext3 noatime,defaults
/home/kzak /dev/mapper/kzak-home ext4 noatime,defaults
or convert UUID to mountpoint:
$ findmnt -o TARGET --noheadings UUID=f1cd38fa-c887-4ab8-834b-c8ee659b97fe
/boot

Wednesday, May 12, 2010

4096-byte sector hard drives

Maybe you're already read some blogs/articles about new 4KiB disks and Linux. These articles usually share one important thing -- WDxxEARS hard drives. Unfortunately, it seems that WDC made a brown-paper-bag bug here. The disks report 512-bytes physical sector size instead of 4096...
hdparm -I /dev/sdb

ATA device, with non-removable media
Model Number: WDC WD15EARS-00Z5B1
Serial Number: WD-WMAVUxxxxxxx
Firmware Revision: 80.00A80
[...]
Logical/Physical Sector size: 512 bytes
Fortunately, it seems that newer models correctly report 4K sectors:
ATA device, with non-removable media
Model Number: WDC WD15EARS-00Z5B1
Serial Number: WD-WMAVUxxxxxxx
Firmware Revision: 80.00A80
[...]
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
Logical Sector-0 offset: 0 bytes
Now the good news. Fedora-13 and all related upstream projects are ready for 4096-byte sector disks. The libparted, fdisk, mkfs.{ext[234],xfs,gfs2} and cryptsetup (upstream, Fedora-14 and RHEL6) have been enhanced to use the new I/O topology to properly align things on the devices.

The I/O topology (aka "I/O limits") is supported since kernel 2.6.31. The topology is exported to userspace by sysfs, for example:
  $ cat /sys/block/sdb/queue/physical_block_size
4096
$ cat /sys/block/sdb/queue/logical_block_size
512
$ cat /sys/block/sdb/queue/optimal_io_size
32768
The kernel also supports topology ioctls since 2.6.32. The parted, fdisk and mkfs.{ext,xfs} use libblkid to get the topology, but some other tools directly use ioctls. So it's better to have kernel 2.6.32 or .33.

The fdisk(8) command uses 1MiB offset and grain to align partitions by default. So the final partition table is usable on hard drives with 4096-byte sectors by default. It means independently on the disk topology. This is the good news for WDxxEARS users.

If you want to use fdisk(8) then think twice and don't forget that fdisk is a low-level tool. Some fdiks(8) hints:
  • use fdisk from util-linux-ng >= 2.17.2
  • read warnings
  • don't use DOS-compatible mode (for backward compatibility this mode is enable by default, you have to use command 'c' or '-c' command line option to disable DOS mode. Note that for the next major release the DOS mode will be disable by default.)
  • use sectors as display units (command 'u' or '-u' command line option)
  • all default sizes/offsets are aligned to the physical block boundary (e.g. "First sector" dialog always provides aligned default)
  • use +size{M,G} convention to specify "Last sector" (e.g. +5G to create 5GiB partition) then fdiskl aligns the size to physical block boundary
  • don't forget that fdisk(8) always follows your wishes -- it means that if you explicitly define first/last sector number then the partition could be misaligned
  • the 'p' (print) command checks for partitions alignment
For more information about 4KiB sectors read:
https://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
http://people.redhat.com/msnitzer/docs/io-limits.txt

Saturday, March 27, 2010

git-pull from translationproject.org

I don't like localization services (portals) that require a write access to the upstream VCS, because distributed version control systems (e.g. git) are designed for a different type of work flow.

Unfortunately, the access to VCS is a preferred method used by transifex.net (note that now transifex allows to pull .po files from .tar.gz and then the final translations are send by e-mail (!) to the package maintainer).

So... I started to use translationproject.org 3 years ago. And it works. It works very successfully without any automatic interaction with upstream VCS. The final translated .po files are available by HTTP or RSYNC. For example:
  rsync  -Lrtvz  rsync://translationproject.org/tp/latest/$PROJECT/$LANG.po  po
downloads selected .po file to the po/ directory. I usually sync my private GIT repository with the latest PO stuff from translationproject.org few hours before package release. The next step after .po files download is to call git-add (for new .po files) and git-commit. The ideal solution is to commit with the --author= option where the author name is a "Last-Translator" from PO file.

Because I'm lazy, I have a script for this task:
  git-tp-sync <project-name> [<lang> ...]
And here is an example from git-log:
  commit d609d0fb8ba1d2302d8bfd0c85903316af83c402
Author: Clytie Siddall
Date: Mon Mar 22 09:36:51 2010 +0100

po: update vi.po (from translationproject.org)

http://people.redhat.com/kzak/git-scripts/git-tp-sync

Wednesday, February 3, 2010

Highlighted patches inside mutt

Tons of patches are delivered by mailing lists. Unfortunately, there is not a native way how properly highlight patches inside mutt. This problem could be resolved by two different methods:

1) call an external command for diffs. You need something like
  text/x-diff;  cat %s | colordiff;copiousoutput
text/x-patch; cat %s | colordiff;copiousoutput
in your ~/.mailcap and

  auto_view text/x-diff
auto_view text/x-patch
set allow_ansi
in your ~/.muttrc. Unfortunately this method sucks, because a lot of patches are distributed in the body of the mail. Note that the mail body is usually better for patches that attachments (see The Perfect Patch).

2) use mutt highlighting. This method is also not perfect, but I use it for years and I am very happy with it. You need to add
  color body green default "^diff \-.*"
color body green default "^index [a-f0-9].*"
color body green default "^\-\-\- .*"
color body green default "^[\+]{3} .*"
color body cyan default "^[\+][^\+]+.*"
color body red default "^\-[^\-]+.*"
color body brightblue default "^@@ .*"
to you ~/.muttrc. That's all :-)

Friday, January 8, 2010

util-linux-ng 2.17

I released v2.17 this morning (CET). A few numbers about the release:
  • 404 patches
  • 488 files changed, 66816 insertions, 48504 deletions
  • 41 developers
  • 22 developers contributed first time
  • 5 months
Thanks to all contributors!

Release highlights

fallocate
  • this NEW COMMAND is a command line interface to fallocate Linux syscall and allows to preallocate blocks to a file.
unshare
  • this NEW COMMAND is a command line interface to unshare Linux syscall and allows to run program with some namespaces unshared from parent.
wipefs
  • this NEW COMMAND is based on libblkid and allows to remove filesystem or RAID signatures from a device.
libblkid
  • libblkid allows to gather information about block device topology, currently supported methods are:
    • ioctl - supported since kernel 2.6.32
    • sysfs - supported since kernel 2.6.31
    • fallback for DM, MD, LVM and EVMS on old kernels (base on code from xfsprogs/libdisk)

    The topology support is mostly designed for mkfs programs or partitioning tools (already used in mkfs.xfs, mkex2fs, libparted and fdisk)
  • libblkid supports partition tables parsing (currently supported are aix, bsd, dos, mac, gpt, minix, sgi, solaris, sun and unixware). This functionality is designed for mkfs programs, DeviceKits, [k]partx or so.
  • libblkid API documentation is available at http://ftp.kernel.org/pub/linux/utils/util-linux-ng/libblkid-docs/
blockdev
  • supports all new topology ioctls
fdisk
  • aligns newly created partitions to minimum_io_size boundary ("minimum_io_size" is physical sector size or stripe chunk size on RAIDs).
  • supports disks with alignment_offset now.

Complete ReleaseNotes:

ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/v2.17-ReleaseNotes

Tarball:
ftp://ftp.kernel.org/pub/linux/utils/util-linux-ng/v2.17/util-linux-ng-2.17.tar.bz2