Planet FreeBSD

BSDCan Trip Report: Eitan Adler

2013-05-23T13:43:57Z

The FreeBSD Foundation sponsored 7 attendees of BSDCan 2013. The first trip report is from Eitan Adler, a doc committer, who attended the BSDCan Developers Summit. Eitan writes:

I arrived Tuesday night and met Colin Percival at the airport. After dropping off luggage at the university, I met up with some of the other developers.

The first day, I attended the "Netflix and FreeBSD" session run by Scott Long. It was interesting to see what kind of problems users of FreeBSD ran into when running at scale.

For the afternoon working group, I chose to attend the "ports and packages" session. A variety of topics were discussed but the most discussed topic was cross-building ports across both versions and architectures. This is a topic that came up repeatedly in prior
discussion and that would come up again in other working groups, so it was good to know about the latest work in this area.

The vendor summit came next. In the past, the vendor summit focused on kernel work but this one revolved around the user land. This is particularly important to me as I run FreeBSD on my laptop as my primary development machine.

At night I spent some time in the hacking lounge or other shared areas meeting people. It was very nice to be able to meet the people I've been talking to for the past three years.

On Thursday I spent my morning in the "Desktop" session. Getting FreeBSD running well on desktops is critical in attracting new developers in the future. Kris Moore, from PCBSD, spoke a lot about the customizations that they made. I pressed to share the improvements
that could be committed upstream. Other issues discussed were packaging for the desktop and a graphical boot loader for FreeBSD/PC-BSD.

The afternoon session for me was "Documentation": a significant portion of the discussion was about the future print edition of the book and what sections need to be updated and improved. In particular, how we could get more source committers involved in writing documentation. We also discussed how to work going forward with other teams that need access to the documentation (e.g., portmgr and postmaster). We also touched on the FAQ, translations, and the new toolchain. The final topic we discussed was the automated QA and statistics tools we have (and don't have) and how we could improve in that area.

After dinner I did some work at the documentation hackathon. I spent the remainder of the night at the hacker lounge discussing kernel internals with Peter Wemm, Sean Bruno, and others.

Unfortunately, I had to leave prior to the conference itself, but I felt that meeting people at the developer summit was well worth the time spent.

brd’s notes » FreeBSD 2013-05-21 16:58:17

2013-05-21T15:58:17Z

First thing we will need a clone of https://github.com/xaque208/puppet-pkgng into /usr/local/etc/puppet/modules/.

This will be pushed out to the clients as long as: pluginsync = true

For me the next step is to create a manifests/init.pp in the new module directory. This is important to me because I want to sync out a /usr/local/etc/pkg.conf to all my machines so that they point to my internal poudriere repos. So I end up with something like this:

file { "/usr/local/etc/pkg.conf":
        mode => 755,
        owner => root,
        content => "packagesite: http://pkg/91-web/
",
}

Once that is done it is easy to use pkgng packages via:

package { "www/apache22":
        ensure => installed,
        provider => pkgng,
        require => File['/usr/local/etc/pkg.conf'],
}

Improvements to Jail Management via the Warden

2013-05-21T13:36:15Z

Kris has an article in the May issue of BSD Magazine, demonstrating some of the new features in the 9.1 version of Warden, the jail management GUI in PC-BSD.

The magazine is available for free download here and Kris’ article starts on page 17.

PC-BSD Hardware Store

2013-05-21T13:20:00Z

Josh Smith has announced the initial launch of the PC-BSD hardware store. This resource is meant to make it easier to find hardware that has been tested to work on PC-BSD.Â The store itself is here. If you have hardware that you know works and which you don’t see listed, please add it to the Hardware section of the wiki so that it can be included.

bsdtalk226 – FreeBSD and Netflix

2013-05-18T16:16:23Z

Interview during BSDCan 2013 with Scott Long, Alistair Crooks, and David Discher from Netflix.

File Info: 11Min, 5MB.

Ogg link: http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk226.ogg

brd’s notes » FreeBSD 2013-05-17 19:47:19

2013-05-17T18:47:19Z

This is a bunch of links to the tools I talk about in my presenation

Tools:

Collectd: https://collectd.org/

Graphite: http://graphite.wikidot.com/
Nagios: http://www.nagios.org/

Poudriere: http://fossil.etoilebsd.net/poudriere

Config Management:
Salt Stack: http://saltstack.com/
Chef: http://www.opscode.com/chef/
Puppet: http://puppetlabs.com/

Subversion: http://subversion.apache.org/

LogStash: http://logstash.net/
Audit: http://www.freebsd.org/handbook/audit.html

CARP: http://www.freebsd.org/handbook/carp.html

OATH: http://www.openauthentication.org/

Serial Console: http://www.freebsd.org/handbook/serialconsole-setup.html

Generic Resources:
FreeBSD Handbook: http://freebsd.org/handbook
Everything Sysadmin Blog: http://everythingsysadmin.com/resources.html

bsdtalk225 – PC-BSD with Kris Moore

2013-05-17T16:07:07Z

Interview during BSDCan 2013 with Kris Moore from iXsystems. We talk about some of the new features of PC-BSD.

File Info: 12Min, 6MB.

Ogg Link
http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk225.ogg

BSDCan and FreeBSD DevSummit 2013

2013-05-17T13:50:00Z

It's that time of the year again - time for the biggest, bestest gathering of BSD geeks from around the world - BSDCan 2013. It was great to see old friends and new faces, exchange ideas and talk about the bright future.

What keramida said… » FreeBSD 2013-05-16 17:04:58

2013-05-16T16:18:45Z

Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).

Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.

An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.

Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66B| 178B|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1268k|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1135k|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 754k|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2079k|4096B|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 397k|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2620k|4096B|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1506k|8192B|  76M|33.6|

The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.

What we would like to do is convert text like “408B” to just “408″, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:

M-x replace-regexp RET \([0-9]+\)B RET \1 RET

This should just strip the “B” suffix from byte values.

For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.

So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:

\,(* 1024 (string-to-number \1))

The full Emacs command would then become:

M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET

This, and the byte suffix removal, yield now the following text in our Emacs buffer:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 772096|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2128896|4096|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 406528|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2682880|4096|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1542144|8192|  76M|33.6|

Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:

M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET

Which produces a version that has no suffixes at all:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  96468992|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  79691776|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  70254592| 772096|   0 |  103809024|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 118489088|2128896|4096|  66060288|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  60817408| 406528|   0 |  99614720|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 119537664|2682880|4096|  54525952|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  79691776|1542144|8192|  79691776|33.6|

Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:

| time           | cpu | cpu | cpu | cpu | cpu | cpu |      eth0 |    eth0 |  disk |      disk | sda- |
| time           | usr | sys | idl | wai | hiq | siq |      recv |    send |  read |      writ | util |
| 16-05 08:36:15 |   2 |   3 |  96 |   0 |   0 |   0 |        66 |     178 |     0 |         0 |    0 |
| 16-05 08:36:16 |  42 |  14 |  37 |   0 |   0 |   7 |  96468992 | 1298432 |     0 |         0 |    0 |
| 16-05 08:36:17 |  45 |  11 |  36 |   0 |   0 |   7 |  79691776 | 1162240 |     0 |         0 |    0 |
| 16-05 08:36:18 |  27 |  55 |   8 |   0 |   0 |  11 |  70254592 |  772096 |     0 | 103809024 | 79.6 |
| 16-05 08:36:19 |  29 |  41 |  16 |   5 |   0 |  10 | 118489088 | 2128896 |  4096 |  66060288 | 59.6 |
| 16-05 08:36:20 |  28 |  48 |  12 |   4 |   0 |   8 |  60817408 |  406528 |     0 |  99614720 | 76.0 |
| 16-05 08:36:21 |  38 |  37 |  14 |   1 |   0 |  10 | 119537664 | 2682880 |  4096 |  54525952 | 23.2 |
| 16-05 08:36:22 |  37 |  54 |   0 |   1 |   0 |   8 |  79691776 | 1542144 |  8192 |  79691776 | 33.6 |

The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 … \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.

Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.

Filed under: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software Tagged: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software

Binary Packages Are Available Again

2013-05-14T08:00:00Z

Six months have passed since the November security incident which brought the Project's binary package building capacity offline; we are pleased to announce that all services are now restored.