Python 2.7 on the Mac: site-package weirdness

I just upgraded to macOS Sierra (10.12) and have been finding some odd things happening with the search path that is being used to find installed libraries. Unfortunately there isn’t a very good solution at this time but I’m sharing what I’ve found in case it helps someone else.

Apple includes Python 2.7 with macOS and it is used by the system for various things. The general wisdom is to not mess with the system-provided install (and whatever you do don’t remove it unless you like reinstalling the operating system). Because I need to install other libraries that aren’t provided with the operating system I’ve installed the Python 2.7 distribution (at this writing version 2.7.12) from python.org and use that for my development work. That is where the problems start to show up.

There are two ways to install new libraries (called packages) using the pip install tool:
1. The global (global for the Python install) site-packages folder.
2. The user’s own site-packages folder.

Since I’m the only user on my system I prefer to just put everything I need into the global site-packages folder since in my case it is equivalent to the user-specific one.

When the Python interpreter starts up it, through various means, creates a search path that it uses to find other packages that you’ve installed when you try to use them with the “import” statement. You can see this yourself by opening up a Python shell and typing:
1. import sys
2. sys.path

You should see something like:

Python Path
Shows the path Python uses for finding libraries after importing sys and typing sys.path

Notice the paths with prefixed with “/System”? There is our problem. Through the standard path extension system our supposed-to-be-standalone Python environment is now seeing one Apple-provided library we might not be interested in using (PyObjC) and a folder where other libraries might be installed.1

The thing to note here is that those paths are at the end of the full sys.path so they will be searched last. If there is a version of the same library (PyObjC) for instance in one of those earlier paths it will be used instead of the system one. So theoretically I shouldn’t have to do anything but I use PyCharm from JetBrains for my Python development and I don’t like seeing this in my projects:
PyCharm package display

This can be fixed however and here is how to do it:
1. Ignore the problem.
If I wanted to do that I wouldn’t be writing this article.
2. Modify a file called site.py that is responsible for generating the Python sys.path
3. Update the file Apple put in place to add those folders to sys.path.

Let’s look at #2 first because that’s the solution I’ve chosen. site.py is installed here by default:

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7

and can be easily edited. The key is to filter out any lines that start with “/System” once the path has been built. sys.path is just a list and can be easily modified by any Python script. I’ve created a gist here with the code. The change was made at line 548 to remove the system paths from sys.path.

One thing to note about doing this is that this file can be overwritten by another Python install and these changes might need to be made again.

What about #3 above? I don’t recommend doing it because it’s an Apple provided file but it’s an easy change to make as well. The file itself is called Extras.pth and just contains two lines: the two paths we want to remove. It lives here:

/Library/Python/2.7/site-packages

The only thing required is to comment out those two lines.

I hope at some point there is a better way to address this problem but for now there is an easy solution for it. If you’re not like me and don’t care about having the system-provided stuff viewable by your other Python install none of this is an issue. As noted earlier if you have the same version of a library in one of the folders that’s earlier in sys.path it will be used instead.

Here is a list of some other reading on the subject I came across while looking into this issue. Some very interesting background information here.

  1. https://mail.python.org/pipermail/pythonmac-sig/2016-September/thread.html#24152
  2. http://stackoverflow.com/questions/14881205/what-is-the-difference-between-pythons-extras-and-site-packages-directories
  3. https://github.com/pypa/pip/issues/2468
  4. http://stackoverflow.com/questions/4271494/what-sets-up-sys-path-with-python-and-when
  1. I have to assume that Apple did this to enable people to add packages to the system-provided Python without breaking anything but I don’t like that it’s there polluting my own Python install. ↩

HMS Terror found after nearly 170 years

I’ve always been a sucker for the stories of Arctic and Antarctic exploration in the 1800’s and the story of the HMS Erebus and HMS Terror is one I’ve followed for quite awhile. Erebus was found in 2014. Now, they’ve found HMS Terror and in fantastic condition (so great glass windows are still in place). This is a great story.

macOS Sierra Tip # 2: Show hidden files when opening a file

macOS Sierra image

macOS Sierra Tip # 2: Show hidden files when opening a file

This isn’t as much of a Sierra tip vs. a general macOS tip since this existed prior to Sierra but it’s a useful thing to know.

When opening a file via the standard Mac open file dialog:

macOS Open File Dialog

press Cmd-Shift-. on the keyboard which will cause hidden files to appear:

macOS Open File Dialog Hidden Files

macOS Sierra Tip # 1: Select the audio output

macOS Sierra image

Tip # 1: Select the audio output

Back in the old days (in other words, the last release of macOS, El Capitan) it wasn’t quite straightforward how to direct the audio on the Mac to somewhere other than the built-in speakers. In that release, assuming you set the preference to show the volume control in the menu bar, you could option-click and get a menu of choices such as AirPlay-compatible speakers.

No More! With the release of Sierra, Apple has decided to unhide the destination menu and bring it front and center. If you don’t have the volume control icon in your menu bar see the instructions below. If you do simply click on it and choose your destination.

If you don’t have the volume icon in your menu bar…

If you want to display the volume icon in your menu bar it’s easy to find. Open System Preferences and click on the Sound icon:

There is a checkbox at the bottom of that preferences panel to show the volume control in the menu bar. Click that and you’re good to go:

What’s Old is New Again (back on WordPress for the nth time)

I seem to have a problem: I can’t seem to make up my mind if I should blog or where the blog I can’t make up my mind to maintain should live. I started on WordPress. Then I switched to Squarespace. I got tired of the limitations in Squarespace then went back to WordPress.com. Then I got tired of the limits of WordPress.com and did a self-hosted install of WordPress running on a Digital Ocean droplet. Then at some point I got tired of having to maintain that myself and gave Squarespace a spin again and was thrilled with it. It had come a long way since the last time I had tried. But inevitably I got tired of the limits of Squarespace again and now I’m back on WordPress.com.

Don’t get me wrong, Squarespace is an amazing platform to keep a personal website and the pricing is phenomenally cheap for what they give you but I had three major issues with it:

  1. Their Markdown editor sucks.
  2. They no longer have an API for external publishing tools.
  3. Their idea of image handling is ridiculous.

Markdown

It’s hard to describe just how poor of a Markdown implementation they have implemented. The editor is buggy beyond belief and it only handles bare bones Markdown. I was not the first one to request a more advanced flavor of Markdown, and I’m sure I won’t be the last, but the requests seem to have fallen on deaf ears because there has been no movement on that part of the tool in a long time. I write everything in Markdown and my preferred flavor is GFM (GitHub Help, or something similar like PHP Markdown Extra because I frequently use the extra items provided by those implementations like footnotes and tables.

Lack of API for publishing tools

I can see the reasoning behind ditching their old posting API because of the way Squarespace pages are built. Every page is usually made up of multiple blocks so that would make creating an external API pretty difficult. It wouldn’t be impossible of course…they could easily limit the API to publishing simple posts (as they seem to do with their own iOS tools) but they obviously think it isn’t worth the cost. I need the ability to post from other tools though and I want the ability to use automation tools like IFTTT with my website. That just isn’t an option with Squarespace right now and from the looks of it won’t be an option anytime soon.

Image handling

Squarespace can certainly afford to take a look at how WordPress (either self-hosted or on WordPress.com) handles a media library. Every image uploaded to WordPress is easily managed using their media library tools. Squarespace, by contrast, offers no such tools. When you attach images to a page or post they are essentially tied to that page and there is no central place to manage them. Want to get a list of every image asset on your site? Good luck with Squarespace because they don’t have a way to see that type of information.

If Squarespace fixes these things I might consider switching back again. Their pricing and feature set for most people is superior but they just have lousy support for the items that are the most important to me. I might have to dump more money into WordPress to get what I want but at least it is an open environment that has far more advanced management tools.

Using tools that inspire us

Courtesy of Dustin Lee/Unsplash.com Courtesy of Dustin Lee/Unsplash.com

I have a confession to make: I’m an iOS app addict. More specifically I’m a productivity app addict (well, photography/graphics too but that would be another post). Todo list apps, calendar apps, writing apps, “glue apps” (like Workflow) – if a productivity app gets good reviews from sources I trust I’m all over it to see how it would fit into my workflows.

Over the last several months I’ve read a lot about Ulysses, yet another writing app. Almost every day I came upon someone extolling the virtues of the new universal version of Ulysses for iOS. A fancy text editor that uses Markdown? It was hard not to hit the buy button but at $20 it isn’t exactly an impulse purchase. I then read that Federico Viticci of MacStories had started using it for his writing.1 I bought a subscription to Club MacStories just to read his impressions and after reading that article I gave in and spent the $20 dollars to buy Ulysses.

As always Federico is thorough but the one part of his article that really struck me though was this:

Drafts, too, is a solid app, and I’m happy that I got to know it better, but, inside me, I wasn’t sure about it either. I could get my writing done in the app, but I wasn’t feeling it. I don’t know if this makes sense to other writers, or if I have a problem, or if it’s a combination of both, and all of us who want to get some writing done on iOS have a bit of a problem with the App Store and abundance of choice. Maybe we are all slightly obsessive about this software thing and we just like to create problems for ourselves.

I think Federico hit on something important here: the feeling of using something. I’ve also found that “feeling it” when using a piece of software is important. I’m sure it’s not important to everyone, but I’d bet a good portion of the people who buy apps like Drafts or Ulysses want to feel engaged with the software they are using. Many crave good design in software; some applications have it, some don’t. I love journaling in Day One on the Mac and iOS. I tried using MacJournal, which is also a fine app in it’s own right, but I just never felt inspired when I sat down to write. Something about Day One inspires me and makes me feel like writing in it. I’d say the same thing about my iOS text editor of choice Editorial. It just inspires me in a way that other text apps have not.

In Federico’s example of Drafts (which I also own and use) he talks about not being inspired. That isn’t a knock against Drafts, it is a very well thought-out application with tons of functionality. It was the first text app on iOS to bring a lot of automation to the table to let it interact with other applications. It doesn’t inspire me to write either.

Look at the byline on the website: “where text starts”. Drafts is designed to take notes and small bits of text and send them on their way elsewhere using the automation features. Yes you can write as much text as you want there and it will work but it isn’t designed for long form writing. That’s where apps like Editorial and Ulysses come in.

Federico is wrong though when he says we are creating problems for ourselves. The guy is an app fanatic to be sure. He works around a lot of shortcomings in iOS using apps like Workflow to accomplish things that, in all honesty, could be done easier on a Mac. But he doesn’t feel inspired on the Mac, he feels inspired on iOS. It’s the platform that works for him and he’s done some amazing things with the scriptability of apps like Editorial or gluing apps together using Workflow and Pythonista. Is there anything wrong with that? Absolutely not. Has he spent a lot of time not writing to accomplish these things? Sure but he’s also learned, and taught, a lot as well. There is a lot of value in those things, just as much as in the writing itself.

I’ve read in the recent past that we should all just settle on a set of tools and stop tinkering with our productivity workflows and to a certain extent I believe that. But I am, and have always been, a tinkerer and I derive a lot of pleasure out of it. Maybe it’s time we stop beating ourselves up over it and just have fun. Maybe we won’t be quite as productive as we could be if we just stuck to a set of tools and got to work but it wouldn’t be as fun. There is a lot to be said about having fun.

  1. If you don’t know of Federico he’s written a fantastic amount about writing on the iPad, including an ebook about Editorial. ↩︎

Thoughts on the season 6 finale of “The Walking Dead”

Just a couple of quick thoughts here because my opinion of the episode matches many others who have already written much better synopses of the show.

  1. Why was the episode 90 minutes long? To sell more commercials? They basically drove around in an RV for 80 of them.
  2. What the hell is with the total transformation of Carol from a badass to a sniveling wuss? After so much character development this is the best the writers could do?!?
  3. Why oh why did they end with a cliffhanger? Guys this episode pissed me off enough that I’m considering not even coming back for season 7. There are many other quality shows out there to watch now. Don’t lob softball reasons at me as to why I shouldn’t watch yours anymore.

Just a terrible end to the season even if the last 10 minutes was a bit brutal. Hell, even the second half of the season itself was getting to me. I get the violence but it is starting to get a bit gratuitous. I get all the gore in the zombie death scenes but they could do so much more with this show than just making it about groups killing each other.

The A.V. Club has a great write-up. Go check it out.

2Do – Restoring Archived Lists

I recently switched from OmniFocus to 2Do for my task management needs and found one thing a bit odd – archived lists still affect the naming choices of new lists. I had a list called “Home” that I had archived. All of the tasks in that list I had tagged with the tag “home” and when I went to save a search I was not able to name the new smart list “Home”, it kept being renamed “Home 1”.

If you are running into the same issue don’t despair, there is a way to “unarchive” those lists :)

  1. Go to the app settings and select Privacy.
  2. Tap Archived Lists.
  3. Slide the switch to any list you want to reappear in Lists on the main screen.

The 2Do developer has made some…um…interesting design choices but thankfully over time is correcting those (erroneous) choices. Hopefully at some point there will be true archiving of lists.

Posting Markdown to WordPress from Editorial

Since switching to Squarespace from WordPress I have not had much of a chance to use my posting script for the Editorial text editor for iOS. I received an email today though from Frank van Exter asking for assistance in using the workflow. Frank was very gracious and it was a nice opportunity to help someone use something I created.

Unfortunately my posting to the Editorial workflow site didn’t quite explain things clearly enough. Here are the two most important things to configure when using this workflow:

  1. Make sure the user that you are plugging in to the workflow has posting access to the blog you’re configuring.
  2. Enter your username and URL into the section of the workflow labeled “Get Connection Details”, not the variables at the beginning of the workflow.

A couple of things to note:

  1. The URL needs to have “/xmlrpc.php” appended to it.
  2. If you are using 2-factor authentication you need to generate a password for the workflow to use in the WordPress admin site.

Also, if you have to reset your password that is done by doing the following:

  1. In “Get Connection Details” slide the Reset Password switch to on.
  2. Run the workflow to reset the password.
  3. Run the workflow again to enter a new password.
  4. Go back into “Get Connection Details” and turn off the Reset Password switch.

Enabling Query Builder and boolean searching in Microsoft Outlook

Why aren’t searches with advanced boolean logic available by default in Outlook?

I hate to say this but it almost appears that Microsoft has abandoned Outlook. Oh sure there are updates every Office cycle but it would seem those are mostly look-and-feel changes.

How about easily being able to create mailing lists by dragging names to a list? Nope.

How about easy tagging? Nope. Sure we’ve got categories but those are cumbersome to use and require a special user interface to set up.

How about stuff like saving searches? Oops. Nope can’t do that either. You can do advanced boolean searches via the search box but you can’t save them.

Want to create a boolean search via the Search Folders feature? Sorry. That is another thing that isn’t doable. Well, not doable out of the box anyhow. In 2016 you can get this feature, but you’ve got to modify the registry. Really Microsoft?

Here is how to do it.