Raspberry Pi Adventures Pt. 1

I got a Pi 2 for Jonas (7) and me to continue programming on–I should blog about that at some point.

Anyway, I’m working through some “fun” issues I’m running into on the Pi, and I figured I should probably blog these.

Issue #1: Display Settings does not let you choose 1920×1080, despite the fact that your monitor supports it.

You have to edit /boot/config.txt to disable overscan. Spoiler: NOOBS adds an extra line at the end of config.txt to force overscan and set some generous borders. 😦

Comment those out too, and reboot, and you should be in business.

Issue #2: Mouse is sluggish and Keyboard and Mouse Settings don’t seem to make any difference.

After some searching, I found https://github.com/raspberrypi/linux/issues/642. I too have a Microsoft Wireless Mouse 5000, and needed to add usbhid.mousepoll=0 to /boot/cmdline.txt

The Pi feels so much faster, now! (It’s funny what a responsive mouse cursor will do.)

Issue #3: System clock displays 24 hr format.

Right-click on the clock and choose Digital Clock Settings. Change %R to %r.

… boy. It’s 11 pm already. Time sure does fly when configuring Linux.

[Sorry this blog post sucks; I just want to force myself to write this down somewhere before I forget.]

ES6 Quicksort

Playing around with some ES6 features:

function quicksort([head, ...tail]) {
   if (head === undefined)
      return [];

   return [...quicksort([for (x of tail) if (x < head) x]),
      head,
      ...quicksort([for (x of tail) if (x >= head) x])];
}

Destructuring, rest, spread, comprehensions… I’m looking forward to writing more ES6.

Hello Roslyn

using System;
using Roslyn.Compilers.CSharp;

namespace HelloRoslyn
{
  class Program
  {
    static void Main()
    {
      string program = Syntax.CompilationUnit(
        usings: Syntax.List(Syntax.UsingDirective(name: Syntax.ParseName("System"))),
        members: Syntax.List<MemberDeclarationSyntax>(
          Syntax.NamespaceDeclaration(
            name: Syntax.ParseName("HelloRoslyn"),
            members: Syntax.List<MemberDeclarationSyntax>(
              Syntax.ClassDeclaration(
                identifier: Syntax.Identifier("Program"),
                members: Syntax.List<MemberDeclarationSyntax>(
                  Syntax.MethodDeclaration(
                    returnType: Syntax.PredefinedType(Syntax.Token(SyntaxKind.VoidKeyword)),
                    modifiers: Syntax.TokenList(Syntax.Token(SyntaxKind.StaticKeyword)),
                    identifier: Syntax.ParseToken("Main"),
                    parameterList: Syntax.ParameterList(),
                    bodyOpt: Syntax.Block(
                      statements: Syntax.List<StatementSyntax>(
                        Syntax.ExpressionStatement(
                          Syntax.InvocationExpression(
                            Syntax.MemberAccessExpression(
                              kind: SyntaxKind.MemberAccessExpression,
                              expression: Syntax.IdentifierName("Console"),
                              name: Syntax.IdentifierName("WriteLine"),
                              operatorToken: Syntax.Token(SyntaxKind.DotToken)),
                            Syntax.ArgumentList(
                              arguments: Syntax.SeparatedList(
                                Syntax.Argument(
                                  expression: Syntax.LiteralExpression(
                                    kind: SyntaxKind.StringLiteralExpression,
                                    token: Syntax.Literal("\"Hello world\"", "Hello world")
                                  )
                                )
                              )
                            )
                          )
                        )
                      )
                    )
                  )
                )
              )
            )
          )
        )).Format().GetFullText();

      Console.WriteLine(program);
    }
  }
}

Reading large xml files

I’m a huge fan of System.Xml.Linq or “LINQ to XML”. However, some documents really are just too large to efficiently process with an in-memory representation like XDocument. For such documents, we need to consume the xml with a streaming XmlReader instead.

As much as I love System.Xml.Linq, that’s how much I hate XmlReader. I don’t know why it is, but every time I have to use an XmlReader, I have to go back to the documentation. And working with an XmlReader rarely feels fun.

At work (by the way, we’re hiring all kinds of developers), we’ve written some really nice code to make reading xml easier. But I’m not at work, and I wanted to process a large set of xml data—namely, the Project Gutenberg catalog in RDF/XML format. So I came up with a simple, efficient solution that I want to share.

The Project Gutenberg catalog data looks something like this:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:dcterms="http://purl.org/dc/terms/"
         xmlns:cc="http://web.resource.org/cc/"
         xmlns:pgterms="http://www.gutenberg.org/rdfterms/">

    <cc:Work rdf:about="">
        <cc:license rdf:resource="http://creativecommons.org/licenses/GPL/2.0/" />
    </cc:Work>

    <cc:License rdf:about="http://creativecommons.org/licenses/GPL/2.0/">
        <!-- cc:license children omitted -->
    </cc:License>

    <rdf:Description rdf:about="">
        <dc:created>
            <dcterms:W3CDTF>
                <rdf:value>2010-01-05</rdf:value>
            </dcterms:W3CDTF>
        </dc:created>
    </rdf:Description>

    <pgterms:etext rdf:ID="etext14624">
        <dc:publisher>&pg;</dc:publisher>
        <dc:title rdf:parseType="Literal">Santa Claus's Partner</dc:title>
        <dc:creator rdf:parseType="Literal">Page, Thomas Nelson, 1853-1922</dc:creator>
        <pgterms:friendlytitle rdf:parseType="Literal">Santa Claus's Partner by Thomas Nelson Page</pgterms:friendlytitle>
        <dc:language><dcterms:ISO639-2><rdf:value>en</rdf:value></dcterms:ISO639-2></dc:language>
        <dc:subject><dcterms:LCSH><rdf:value>Christmas stories</rdf:value></dcterms:LCSH></dc:subject>
        <dc:subject><dcterms:LCC><rdf:value>PZ</rdf:value></dcterms:LCC></dc:subject>
        <dc:created><dcterms:W3CDTF><rdf:value>2005-01-06</rdf:value></dcterms:W3CDTF></dc:created>
        <dc:rights rdf:resource="&lic;" />
    </pgterms:etext>

    <!-- etc. -->

</rdf:RDF>

Let’s first look at the wrong way to read this data:

static void Main()
{
    XNamespace nsGutenbergTerms = "http://www.gutenberg.org/rdfterms/";
    XNamespace nsRdf = "http://www.w3.org/1999/02/22-rdf-syntax-ns#";

    XDocument doc = XDocument.Load("catalog.rdf");
    foreach (XElement etext in doc.Root.Elements(nsGutenbergTerms + "etext"))
    {
        string id = (string) etext.Attribute(nsRdf + "ID");
        string title = (string) etext.Element(nsGutenbergTerms + "friendlytitle");

        Console.WriteLine("{0}: {1}", id, title);
    }
}

A couple of problems:

1. speed—the program sits around for 5 seconds or so before outputting anything, while it loads the 128MB xml file into memory.
2. memory usage—loading the 128MB file pushes the memory usage from 10,328K to 731,832K (as reported in task manager). I don’t want to read too much into that value, but we can certainly agree that loading the whole file into memory at once isn’t optimal.

This is the worst of both worlds: the program is slower than it needs to be, and it uses more memory than it should.

… but did I mention that I love LINQ to XML? Processing each etext element as an XElement instance is really convenient.

Ideally, we would want to combine the efficiency of reading the large xml file with an XmlReader with the convenience of handling each etext element as an XElement instance.

Cue Patrick Stewart saying, “Make it so”:

static void Main()
{
    XNamespace nsGutenbergTerms = "http://www.gutenberg.org/rdfterms/";
    XNamespace nsRdf = "http://www.w3.org/1999/02/22-rdf-syntax-ns#";

    using (XmlReader reader = XmlReader.Create("catalog.rdf",
        new XmlReaderSettings { ProhibitDtd = false }))
    {
        // move the reader to the start of the content and read the root element's start tag
        //   that is, the reader is positioned at the first child of the root element
        reader.MoveToContent();
        reader.ReadStartElement("RDF", nsRdf.NamespaceName);

        foreach (XElement etext in reader.ReadElements(nsGutenbergTerms + "etext"))
        {
            string id = (string) etext.Attribute(nsRdf + "ID");
            string title = (string) etext.Element(nsGutenbergTerms + "friendlytitle");

            Console.WriteLine("{0}: {1}", id, title);
        }
    }
}

Apart from noticing the similarity between this and the previous code block, the most interesting part of this code is the ReadElements extension method.

Before calling ReadElements, the code positions the reader on the first child of the root element. Then, ReadElements is called with an XName referring to the etext element. All of the etext elements are returned as a sequence.

This is exactly what I want: the program starts processing etext elements nearly instantly, and the memory utilization is barely noticeable.

Let’s look at the implementation of ReadElements:

/// <summary>
/// Returns a sequence of <see cref="XElement">XElements</see> corresponding to the currently
/// positioned element and all following sibling elements which match the specified name.
/// </summary>
/// <param name="reader">The xml reader positioned at the desired hierarchy level.</param>
/// <param name="elementName">An <see cref="XName"/> representing the name of the desired element.</param>
/// <returns>A sequence of <see cref="XElement">XElements</see>.</returns>
/// <remarks>At the end of the sequence, the reader will be positioned on the end tag of the parent element.</remarks>
public static IEnumerable<XElement> ReadElements(this XmlReader reader, XName elementName)
{
    if (reader.Name == elementName.LocalName && reader.NamespaceURI == elementName.NamespaceName)
        yield return (XElement) XElement.ReadFrom(reader);

    while (reader.ReadToNextSibling(elementName.LocalName, elementName.NamespaceName))
        yield return (XElement) XElement.ReadFrom(reader);
}

The documentation comments should be pretty self-explanatory, but it’s probably important to call attention to the side effects: ReadElements expects an intentionally positioned xml reader. Once ReadElements is done returning XElements, the reader will be positioned at the end element of the initially positioned element’s parent.

I should also point out it would be trivial to add an overload of ReadElements that didn’t take an XName and simply returned a sequence of the initially positioned element and all of its following siblings. But I don’t need that method yet, so I didn’t write it.

ReadElements will certainly allow me to process this large xml file more efficiently and easily than exclusively using either an XDocument or an XmlReader. Hopefully this method will be helpful to some of you, too.

Google Reader Tip

Sorry to break a long hiatus with such a lame, non-programming post. But, here’s a tip for Google Reader users:

A coworker of mine appears in my google talk list, but was not able to see my shared items in his Reader. We checked the settings a number of times, but to no avail.

Another coworker recommended the “try changing this, and then changing it back” strategy. I was skeptical, but reluctantly tried it…

And it worked!

So, if you use Google Reader and share items, check to make sure you’re sharing with everyone you intend to:

1. Click sharing settings on the left-hand side of Reader:

2. Click “Change” on the right-hand side of the newly displayed page:

3. Toggle the radio button to “Share with ‘Friends’” and click Save:

Repeat steps 2 and 3, but select “Share with all of my chat contacts,” the second time.

Hope this helps.

C# compiler eccentricity of the day: throwing lambda

Here at work (gratuitous link; oh yeah, and we’re hiring), we have a Verify helper class. Verify lets you succinctly validate (or verify, if you will) various method invariants. For instance, non-nullness:

Verify.IsNotNull(name);

The problem with these helper methods is that when an invariant is violated, an exception is thrown far from the actual affected bit of code. Always moving one frame up the call stack to see the real code that’s failing quickly gets annoying.

I started thinking about how to mitigate this problem, and realized that with Visual Studio’s excellent debugger support for delegates, the call site could include an Action<Exception> that just threw (throw-ed?):

Verify.IsNotNull(name, ex => throw ex);

Except that doesn’t compile.

Wrap the lambda body in curly braces (don’t forget the extra semi-colon) and everything works as expected (including the debugger breaking within the lambda!):

Verify.IsNotNull(name, ex => { throw ex; });

But unfortunately that adds so much syntax that the “solution” is more annoying than the problem I was initially trying to solve.

Has anyone run into anything like this before? Does it make any sense why that statement wouldn’t be valid as a lambda expression?

C# reminder of the day

Static data is not shared among constructed generic types.

That is, the final line of output from the following program:

using System;

class Program
{
    static void Main()
    {
        NonGeneric.PrintCount(); // "Called 1 time."
        NonGeneric.PrintCount(); // "Called 2 times."

        Generic<int>.PrintCount(); // "Called 1 time."
        Generic<string>.PrintCount(); // ?
    }

    public static void DoPrintCount(int count)
    {
        Console.WriteLine("Called {0} time{1}.",
            count, count > 1 ? "s" : "");
    }
}

class NonGeneric
{
    public static void PrintCount() { Program.DoPrintCount(++count); }
    static int count;
}

class Generic<T>
{
    public static void PrintCount() { Program.DoPrintCount(++count); }
    static int count;
}

Is “Called 1 time.”

Strange Framework design decision of the day…

Today I encountered the strangest .NET Framework design decision I’ve seen in recent times:

HashSet<T>’s GetEnumerator method returns a public struct HashSet<T>.Enumerator.

Let’s count how many Framework Design Guidelines this violates:

1. Avoid publicly exposed nested types.

• violation: duh.

Do not define a structure [instead of a class] unless the type has all of the following characteristics [including]:

2. It is immutable.

• violation: calling MoveNext mutates the enumerator object.

3. It will not have to be boxed frequently.

• violation: passing a HashSet<T> as a parameter to a method that accepts IEnumerable<T> (Linq, anyone?) will hide the class’ GetEnumerator method. Therefore, any calls to GetEnumerator call the interface method which requires boxing the HashSet<T>.Enumerator to return an IEnumerator<T>.

4. [Any others you see? Leave a comment.]

 

I really want to hear the arguments in favor of the shipping design.

.Any overload

Nice:

if (folderPaths.Any(path => !Directory.Exists(path)))
    throw new DirectoryNotFoundException();

Nicer:

string pathNotFound;
if (folderPaths.Any(path => !Directory.Exists(path), out pathNotFound))
    throw new DirectoryNotFoundException("Could not find path: " + pathNotFound);

The implementation of

    public static bool Any<T>(this IEnumerable<T> source,
        Func<T, bool> predicate, out T found)

left as a trivial exercise to the reader.

Hello, world!

Jonas Adam Carpenter; born June 4th @ 6 lbs. 3 oz., 19 in.

(Sorry for the entirely non-technical post. I’ll be blogging more C# again, soon.)