Sep 052014
 

I received an email from Alex Lang and he asked about my current use of Electronic Lab Notebooks

Hi Dave,

I’m a physics graduate student who just started using a WordPress based ELN. I really found your thoughts on ELNs helpful, especially:
http://www.davelunt.net/evophylo/2009/03/wordpress-as-eln/
https://speakerdeck.com/davelunt/electronic-lab-notebooks-for-ug-students

Since someday I want to be a PI, I had some questions for you. If you would prefer to answer as a blog post, that would be fine by me.

I wondering if you could elaborate more on the mechanics of how you actually implement ELNs with your students. For example, some questions I had are:

Do students have an ELN on your website? Or do they host their own?
What happens when students leave the group? How do you keep a record of the ELN?
Does the whole lab share ELNs with each other? Or do you restrict it to you seeing students ELNs?
Do you ever share parts of the ELN with collaborators / outside people? Does that work out?

Thanks again for the insight into ELNs!

Alex

Hi Alex, thanks for the prompt to write something

Even though it was quite a while ago that I wrote my posts about ELNs most of it still holds for me. The post you link to above was talking about graduate students and postdocs, whereas the slides refer to undergraduates doing a short project but actually both are implemented in a similar way.

Do students have an ELN on your website? Or do they host their own?
What happens when students leave the group? How do you keep a record of the ELN?

I can answer these two together. I set up a blog ELN for all, and nobody sets up their own. Two reasons for this: Firstly, setting up a blog can be intimidating if you’ve never done it before, as many starting an ELN have not. Secondly, the blog is owned by me and can’t easily be deleted. This ensures I always have the experimental record even if researchers leave. I think this is important. Of course the researcher has admin rights and can always save a copy when they leave and take it with them, but I always retain the ELN record.

I host all ELNs on WordPress.com. The reason is that I trust them more than I trust my personal domain. You will have a reduced list of themes and plugins that can be used, but it is fast and robust.

Does the whole lab share ELNs with each other? Or do you restrict it to you seeing students ELNs?

Actually, only the project supervisor (me) usually sees the ELN. This is not really a decision, just how it worked out. I would be happy to let anyone in the lab see, and the people writing the ELN probably wouldn’t mind either, but nobody really wants to read the experimental minutiae apparently. We have regular lab meetings and this provides all the details most people want. Sometimes I will add a postdoc and a PhD student onto each other’s ELN if they are on similar projects, but generally people are happy without seeing details. In some ways I think this is a shame, we can all learn something from seeing how others do science and write up the record. I may try to change that and put everyone on every ELN by default.

Do you ever share parts of the ELN with collaborators / outside people? Does that work out?

I have shared an ELN with another co-I on a grant. It worked well, though it was not a major source of info for them (I don’t really know how often they read it). They preferred meetings where the researcher would summarise and discuss rather than reading the experimental record (which is sometimes a bit dry). Other co-Is have not really wanted to even see the ELN. I however quite enjoy enjoy browsing new work by my people and I am excited when I get a new post notification!

This brings me to something I hadn’t really thought much about before. Use of ELNs is not primarily a technology issue, it is a personality issue. If you don’t want to read a paper notebook, you won’t want to read an electronic one. Even if you don’t want to read other ELNs you should still keep one yourself:

  • It will help you in writing your manuscript. Some descriptions and methods will already have been written and only require copy/paste. I recently saw an excellent ELN post by my postdoc Amir that was a manuscript draft. Just explaining in the ELN what had been done and what conclusions could be drawn had created that first manuscript version.
  • It is more robust. Your leaking ice bucket cannot ruin the whole year’s experimental record. There is version control. It is backed up in the cloud and if you are wise it has a regularly saved local copy too.
  • The features of a WordPress ELN make it powerful. Search, tags, and categories make my day much easier and more productive.
  • I think it is just easier to keep information this way; easier to paste in text, and screenshots, and protocols, and web links. Even for people working at the bench rather than the computer I think it is easier.

My personal ELN is still very successful I think. I don’t research every day which makes the search function vital. I have had some minor failures but it is the best experimental record I have ever kept. My failures have taught me about the value of records with lots of searchable tags, the importance of explicit data file versions, and never to scribble something to ‘type up later’. The times the ELN gets flaky is when I’m too impatient and do the next thing before really creating a record for the last.

I have 2 students starting later this month, and I will set up a WordPress blog ELN for each. If anyone would like to add their views and experiences please leave comments, or email me.

 

Mar 072009
 

In my previous post I said that I needed to find out about associating data files with electronic lab book entries. I think I’ve got it sorted out now. Imagine you are recording an experiment and want to associate the actual data files with the post of your ELN blog. I don’t mean put the actual text there, rather a link to the file of whatever format.

In the post toolbar Upload/Insert area click on the last icon “Add Media”. You should then be in the “Add files from your computer” area. This doesn’t work for me and I have to click on the link that says “browser uploader” instead. Select the file from your computer and upload it. When it has uploaded you will see the details screen. Give the file an informative name as this will be the link text inserted into your blog post by default. If nothing much happens when you click “insert” then make sure you have clicked on the “File URL” button. This will add a link to download the file into your post. The actual files themselves are held in

domainname/wp-content/uploads (then year and month folders).

These data files are not backed up if you ‘export posts as xml’ as your backup strategy, but its easy enough to ftp a copy weekly, or even get a script to do it.

I normally fill my ELN with methods and conclusions and things to do next. I keep the data files and results archived locally and try to include the file name and location in the notebook (if I remember). Now I’ll move to keep both together in WordPress. It feels a much better arrangement, and I hope I will spend less time looking for the exact input file that generated a specific tree.

Mar 032009
 

In my previous post I said that I had been testing software and decided that the new version of WordPress would be my electronic laboratory notebook (ELN). This post details some of the reasons.

WordPress is open software under a GPL license, allowing it to be distributed and modified. I really like this. There is a very large and active community of people supporting it. This means that even if WordPress HQ was hit by a meteorite tomorrow the software would be unlikely to die. Like Evernote (that I mentioned previously) WordPress exports as xml, but in a standard format that can be easily imported into Blogger or other software. It can also be made to export posts as PDF and MSWord via plugins. I can’t think of anything more important than having my data in easily interchangeable formats, I have changed my mind about ELNs before and I don’t want to be locked in.

Syncing: Since blogs are ‘on the web’ they are accessible from anywhere and cannot be out of sync.
Style: WordPress2.7 is very slick and attractive software indeed. More importantly since content and style are separate entities you can swap between >600 free themes. Yes, I know it still sounds superficial to be so worried about how it looks, but you will have to live with your ELN for years! I need flexibility to control appearance and layout. Beyond appearance WordPress2.7 has a lot of flexibility to control different aspects of your environment.

Other aspects of WordPress
Plugins: There are over 4000 community developed plugins to extend functionality. If its not there you could write it yourself. These include several functions I’ve become interested in using:- exporting posts as PDFs for archiving the last few months activity, automatically colouring code snippets, posting from pre-prepared templates.
Comments: I’m thinking primarily about my own use at the moment, but would like to use this with students and postdocs too. Sharing and commenting is very valuable.
Versioning: WordPress2.7 has a good, wiki-like versioning system. This is useful if you update an experiment and need to see exactly what changes you, or a lab-member, has made. Here are a couple of screenshots.

Privacy: This system allows you to implement your own level of privacy, from completely open, to selected users, to complete privacy according to your preference. I have thought about this issue quite a bit and implemented a private ELN for myself. If you come to different conclusions it is possible to allow selected users to subscribe, edit, author or administer as required.
Tags and Categories: The combination of categories (“selection”), subcategories (“hemagglutinin”) and tags (“dN/dS”) works better for me than just tags or just hierarchical folders.
“Press This”: the (strangely named) javascript button can be dragged into your browser and can be used to add selected text or the page as a blog post whenever you come across something. It isn’t as good as Evernote’s capture tool, but it captures text fine.
On the move: There is a WordPress application for iPhone/iTouch and other mobile devices. Posts can be emailed in too.

Something I’m not sure about yet is associating (non image) data. If I have an excel spreadsheet that is part of an experiment I believe I can upload it and then link to it from within the blog text. If I’m right that would be useful as it would permanently keep data and description together. The data files would then just be in an ftp-accessible folder on the server which I suppose could be backed up with a script. I’ll have to investigate more and see if this really is the case.

I tried WordPress out with a free blog hosted at wordpress.org. I used this for a while, but wasn’t entirely happy with the flexibility. The themes I wanted weren’t among those available and it isn’t possible to install plugins there. So although it was quite functional I decided to rent my own domain hosted at DreamHost. That is what I’m working with now, and in a post soon I’ll sum up my experiences both good and bad. Overall I’m quite happy with it though.

Mar 032009
 

Last year I posted about my search for an electronic laboratory notebook (eln). Since then I’ve had experience of several systems, and my ideas have changed quite a bit, so I thought it was time for another post.

I listed a number of criteria that I thought were important for my eln. I got it wrong however, it turned out that two of the things I valued most weren’t even on my list- synchronization and style.

Synchronization
I was at that time using journaling software, and syncing using either their internal capabilities to link to my .mac account, or later on using the excellent Dropbox to sync. This worked well 99% of the time, but 99% isn’t enough. I wanted my eln to be in perfect sync between work and home and NEVER get corrupted by conflicting changes. So I abandoned these journaling software elns despite the nice software.

Style
This sounds superficial, but I realized that using attractive software is very important to me. I spend most of the day working with my lab book, and I just can’t commit to using something spartan and ugly. I want to use a modern software environment that is attractive and well laid out. I was initially put off blogs and wikis not just because they are generally short of features but because most are really ugly to work with.

(Its not all about sync and style, I want many other things too, but I’ll discuss those specifically in a follow-up posting)

I’m impressed with Evernote, and seriously considered this as my eln. It works very well in fact, yet I have decided not to go this route. Evernote is proprietary software, which isn’t a deal breaker for me, but compared to many it has very poor export. It will export your data as an xml file, but as far as I can determine this xml file cannot be opened by any other application. If Evernote went out of business tomorrow could I get at my information? Well, the local copy would still work with the application but I would not be in a good situation at all.

So, having researched this quite a lot, I have decided that my eln will be WordPress. Not any blog, but specifically the new WordPress 2.7. I’ve been using it for 2-3 months now and I’m going to write why I’ve chosen it, and how I’m using it, in a following post.

Blogs have been discussed and used for eln before. I’m not suggesting a whole new approach here, just recording my personal experiences in finding an eln solution that works for me.

May 062008
 

I’ve been thinking recently about how best to organize my research data and experimental records. I’m not sure my needs are the same as everyone else’s, but I doubt they are so very different. Mostly I want to record GenBank searches, phylogenetic analyses, little bits of perl code. Although I support open data, I choose not to before publication. To be honest I don’t even really understand the need or desirability of open lab books. Maybe I’ve just come across more unscrupulous people?

Here is what I think I need

  1. I want it to be electronic (obviously) as its easier to incorporate my results and code that way. Also I have almost forgotten how to write and typing is much easier.
  2. I want the format to be open, not proprietary. It is equivalent to me if it has good batch export to a common format e.g. RTF. That way I’m not locked in to a particular program or system. I have swapped between several by batch exporting my records as RTF and importing them to another program.
  3. It should have both a chronological listing format and good search functions
  4. It should be easy to insert pictures and resize if necessary.
  5. It should be easy to work with formatted text, headings, tables and lists
  6. Links to other entries are useful, as are tags and a good folder hierarchy.

So I have been thinking about (a) Wikis, web and blog software (b) MS Word and Google docs (c) Specifically-designed electronic laboratory notebooks (d) Journaling software. Here are some thoughts and where I am now.

A. Wikis, Web and Blog software
There are a lot of different wikis out there, some simple, some very powerful. You can set up a free wiki on a public server. OpenWetWare is a good example. I don’t want to go that route as I want to keep my experiments private until I publish. Some positives of wikis are that they can be published locally on your hard drive, or on a lab server. TiddlyWiki is a favourite of mine. Wikis give the possibility of sharing. If I had a student using this approach I could log in and read her lab book whenever I wanted. I’ve tried a few wikis and my feeling is that they are a little more difficult to use than necessary. I am used to the ease and power of word processors and well made software. The ones I have tried don’t have that sort of polish. I don’t really know enough about wikis. Is it possible to batch export entries? If I kept a wiki lab book, then found something better would it be easy to select all, export as (say) RTF and then import into my new favourite software? I suspect it may not be very straight forward. I don’t see wikis as the way to go, it doesn’t feel right for me.

Blogs are great, organized chronologically (just like a paper lab book), with the ability to insert pictures, tables and formatting. Again I worry about the sophistication of blogs. I suppose there are different varieties. But I have many of the same questions as of wikis. I am not yet convinced the software is good enough. Blogger is a bit horrible and buggy. Inserting pictures can be problematic. It doesn’t seem to be up to the standard I would expect of my main information storage system.

There are some excellent open source content management systems available. Joomla! for example is a free and powerful content management system for creating web sites. I think setting up templates for different experiments could be a useful way to go. But same complaints as above.

B. MS Word and Google Docs
The new version of MS Word (2008 on my Mac) has a notebook layout. This looks like it could be just the thing, but it isn’t. Its not well designed and I gave up on it within half an hour. Standard Word docs are a possibility too. But then each experiment would be in a separate document I guess, quite difficult to browse and organize them.
Google docs are a simple set of office applications that you use online. They are interesting as they are stored on Google’s servers. You can also mark a document as shared and select who has access to it. It might be a useful approach to showing your supervisor your work if you are a student.

C. Specifically-designed electronic laboratory notebooks
These are usually commercial, and concentrated on features like security, validation and audit trails of what changes were made when. There is a list of some software here and the wikipedia entry is useful. I don’t find these really meet my needs as I really don’t care about documenting my edits and validating my electronic lab book any more than I did my paper one. It might be different if you work in industry. I recently saw iPad though which is free and definitely worth a look. After testing it a bit though I don’t like the environment very much.

D. Journaling Software
These are applications for organizing databases of data in the form of notes, and other media. They are often very sophisticated and powerful. They tend to be operating system specific. I use OS X (most of the time) so the examples below relate to Macs, but Microsoft and Linux have similar stuff. I have used Journler until recently. This is a really excellent software package (US$35), but I have recently swapped to use MacJournal which costs the same. This is a similar program, not quite as well-designed as Journler but it does have the advantage of syncing with my .Mac account so I have the same data available at home and in the office. It turns out this is really important for me.

These journals are a database of entries. Entries can be formatted like a word processor document with styles and tables and inserted pictures. Tags and search facilities are good. Different types of media (e.g. PDFs, JPG) can be dragged in. Audio (or video) notes can be recorded and added simply. Chronological listing and folder structures are useful. Export of notes as html, PDF, RTF, text etc should ensure that the information is always accessible even if you stop using the software.

I am very impressed with information journals like Journler (see pic below) and MacJournal as electronic lab books. They fit my needs, are powerful and very polished pieces of software. I’ve realized that working in a nice software environment is more important to me than anything else. They don’t have the automatic publish to web strengths of blogs and wikis which I guess can be very useful to coordinate between lab members, or to allow a supervisor to routinely read a student’s book. They both do export as html though and hopefully this aspect will develop.