DCC Roadshow Oxford

Sep 262011
 

Below is a copy of a blog entry I put on the DCC website after attending their Roadshow in September ……

 

As a rather out-of-practice ecologist I was looking forward to the DCC Roadshow with a little trepidation. Would the material be aimed at digital curators writing code to fine tune their repository functionality?  Or would it be of help to someone like me… drafted in from a different area to work for a few months trying to get to grips with the management and sharing of spatial data for the JISC funded IGIBS project.

Well, all concerns were rapidly dispelled as the first day got underway and the programme was delivered with a broad church in mind.  We were accommodated in the rather glorious Wadham college and I began to feel like a well-cared for data object. Securely stored in safe surroundings and my metadata had obviously been carefully read as I was now being professionally transformed into a more up-to-date format just  before my existing file type became redundant.

It  was clear that Oxford had taken data management  seriously and was a centre for developing new ideas and services to aid their researchers and the wider community. Particularly inspiring was a presentation from David Shotton, who had taken his position as Director of the Bioinformatics Research Group to add considerable professional  weight to the importance of developing a professional  data management  infrastructure. He was leading a project to  manage, publish and cite datasets. Using the theme of infectious diseases was the icing on the cake for me. The freeing of data on this subject is providing both an “academic good” and a very obvious “public good”, especially to countries with less ability to access the journals and data than more developed regions  with perversely  less public need for the data.

Themes that came out of the day were the importance of standards in metadata to bridge across disciplines and the need for institutional repositories to hold  the “long tail” data that most researchers produce en route to paper publication.  It wasn’t until later that evening , when I found myself not in the pub but spending an hour watching Bryan Heidorn on YouTube presenting a version of his “Curating the Dark Data in the Long Tail of Science” paper,  that I realised how the day had really got my remaining grey cells up and firing.

Day two comprised mainly of  group work where I began to understand the roles and problems facing many of the different data managers attending the Roadshow. The need for researchers to fully engage with the data management service providers became apparent as the group sessions developed. There are whole armies (well maybe platoons) of professional data experts trying to herd the cat like academic data producers into the pen of good practice. From meeting the Roadshow participants and coming to understand their roles and expertise,  it is hard to understand how there could be a problem with data management.  It was not  until I came back to Aberystwyth’s IGES department and saw individuals trying to finish off theses, and papers as well as prepare for the new academic term that I really understood the problem. As was suggested in day one of the event, it is only the moment when data management arrives at  the top of an individual’s priority list that it actually gets done. I have since been thinking about trying to start a project using “Nudge” theory to try and move data management up the priority tree in an academic department.

Day three was focused on the tools available to aid the  researcher in their data management . The DCC data management planning tool shone out as a valuable asset for  meeting the increasing need of funding bids to meet the demands/requests of the funding councils.  For me it will provide the framework on which to hang my thoughts and conclusions about managing  spatial data and  provides some of the structure for my work on the IGIBS project.

Now back at Aberystwyth and slipping back into my routine, the measure of success, at least from my perspective, for the Roadshow will be both the new sections in my  final report that I feel able to write and  the changes in working practices that come  about as a result of  my attendance.  Well it’s too early to say, but as I have just  moved metadata creation for some  soon to be used images and shape files to the top of my priority tree, maybe the digital curation  my brain received from the DCC is beginning to show.

 

 

 Posted by at 20:02 asides Tagged with: , , ,  Comments Off on DCC Roadshow Oxford

Metadata Are Go!

Aug 122011
 

We have started to put some of our IGIBS data into GeoDoc. We will have a couple of datasets showing land use change particularly for forestry for the whole of the Dyfi catchment . There is also  some more detailed data from vegetation surveys looking at a series of plots from the different habitat types found in the Dyfi Biosphere.  More data will be prioritised for inclusion into IGIBS as we refine the potential uses for the project. I spent a few hours helping with some of the survey on Borth bog so that I  will be able to complete the metadata records with an understanding of the process involved in its collection.  

It has become very  apparent that for many of the geospatial datasets then it is the creator of the data that is best placed (or possibly the only one able) to complete the metadata records.  So thanks to Jonathan Brownett who is now starting to complete what will might just be the first INSPIER compliant metadata record for the project  (may even be the first for the University).

The process really brings two issues to the fore. Firstly that  when the metadata record is complete and published the data can  really  take on a life of their own with the possibility of access by anybody who cares to search for them.  Almost makes the evangelist in me want to start the Data Liberation Army, but I think that the DLA acronym has been taken.   Second is that there could be an immense amount of work needed in the preparation for the INSPIER Directive. If every dataset looked at by IGIBS needed metadata creation then there is probably a year’s  worth of work for one meticulous individual or several  days’ work for many tens academic and post graduate  staff. Either way the importance of metadata creation at the point of data creation must be a consideration  for future datasets. 

I think it will be, shall we say, interesting  to see how INSPIRE is applied to universities and to individual researchers as there will simply not be enough resources to complete INSPIRE compliant metadata for every iota of geospatial information held by a university department. If  universities are to fall under the INSPIRE regulations then some prioritization of  metadata creation and data publication will be needed to give the academic world a fighting chance of  meeting at least the spirit if not the letter of the Directive. I can only assume that it might for example be aimed at newly published and publicly funded research from academic staff as then there will be a number of ways to make data available once it is associated with its published article and held in institutional  or research council repositories.  If, for example, all the data from undergraduate and postgraduate theses and dissertaions   need to be made available then it would be an insurmountable hurdle to jump.

It will be important for institutions to realize that while there may be some extra resources needed to meet any new regulations then each metadata record and collection of data that is made public will be a benefit to the wider academic community and to the the community at large.  So taking an attitude of  “selfish altruism” might be the best way of viewing any future work needed to make geospatial data  resources available for more people.

 Posted by at 12:58 asides, Project Management Tagged with: , ,  Comments Off on Metadata Are Go!

Publish Data or be Dammed

Aug 082011
 

My apologies to the Duke of Wellington for mutating his often quoted call to a jilted mistress about his intimate letters, but the sentiments in the original statement do suggest the power of the publication process to give information a life of its own. 

If geospatial data were published in a similar way to research findings (or even to letters from the rich and famous to their mistresses) then data management and the academic spatial data infrastructure (SDI) would be an even more  rapidly developing entity, that had the commitment of every academic researcher that generates such data.  OK so a sweeping unrealistic statement but this is how I came to the thought…. 

One of the pillars of modern day science is the peer review process. It takes a piece of original research and, through the  publication process, many stages of refinement are applied to it until the researcher is satisfied that it stands a chance of acceptance by an appropriate journal. During this process it will be proof read, checked for errors and formatted in the appropriate way. Then after input from independent referees it will be further improved and finally, if judged acceptable, the research is published where it can be accessed, seen and discussed by the wider scientific community (or any community that wishes). 

After publication it is archived and catalogued so that it can be found on-line or in hard copy and can be used and quoted by anybody who wishes. There may be a network of people and libraries that will have subscriptions to the journals and they will see the newly published articles appear on their shelves, desks or screens every few months. Finally if the research is worthy it may be used as a component of more research and go on to help develop the knowledge base. 

Concomitant with this process is the recognition given to the researcher and to the associated Institution that can result in promotion for the former and extra funding for the latter. This rather idealised description of the peer review process is something that the majority of academic staff and postgraduate students are fully engaged with and committed to. You might have to forgive my simplistic view but all I wish to establish is the principle that the publication process and the recognition it attracts drives the quality, accessibility and reuse of research findings. 

Now let’s consider how geospatial data is managed. It’s not so easy to simplify the process as there will be much greater variation. Some important data will be lodged with data centres where it may have a guaranteed 10 year life span (if its lucky) or maybe the metadata will be put in a discoverable place with a series of hurdles to cross before anybody can get access to the data itself. Quite often the data will never leave the IT systems of the Institution that it was created in; rather it will take second place to the research publications and may not be made accessible at all. This has been shown to be through worries over intellectual property, through a lack of awareness of a suitable data management and publication methods and a lack of recognition for such activities.  One thing is for sure, that most data will not have the same exposure to the science community as the research findings it supported. 

The long term life expectancy of such data is also likely to be shorter than its wordy cousin the research paper.  In fact it may not even exist after its collector has moved posts or retired or suffered a serious IT problem. It is very unlikely to be archived as well as the printed word and its creator is much less likely to have received credit for collecting it and the institution she/he works for is unlikely to receive improved research ratings or extra funding for generating it. 

Now imagine a word where data is King (or at least Queen alongside the research paper King) and research funding and University Chairs are partially reliant on the proper publication of peer reviewed data sets. I think that a fully functional academic SDI with all the bells and whistles that you could want would be a reality within the next decade. In the same way that JANET has, and continues to develop in the UK with its fast speeds and links to other country networks so would the SDI. 

Obviously this isn’t an original idea, there are a few journals dedicated only to data publication and there are strong policy statements all the way from Government through the publicly funded research councils and even to a few Universities that make the publication and accessibility of data a priority. What is missing are real incentives for researchers to treat data in the same way as research findings and until this gulf is filled then data will be the poor relation of the academic publication World. If the translation of the INSPIRE (see my previous post)   directive into European Governments’ actions includes Universities then maybe it will provide some significant infill for this gulf and move the discussion from “why should I?” to “how do I?”

Background

 Posted by at 12:29 asides Tagged with: ,  Comments Off on Publish Data or be Dammed

Shibboleth: report of early stage development using biometric id.

Aug 032011
 

I guess that anyone reading this  will already know that Shibboleth is an open source federated user authentication infrastructure that allows control of user sign-on between or within an organisation/s.  I had a vague memory from childhood Sunday School classes that there was an older meaning to the word apart from the Times crossword type definition of a feature that belies your social or ethnic origins.

So just in case there is a person left who hasn’t Googled “Shibboleth” here are the conclusions from 5 mins of browsing.

The origin is from Hebrew in the  late Bronze age (1100BC). It was used as a test of racial origin to try and filter out a group called the  Ephraimites from the Gileadites. In a nut shell anyone attempting to cross the river Jordan after a particularly nasty battle was asked to pronounce the word Shibboleth (meaning stream or  ear of wheat) . If you were unlucky enough to pronounce it Sibboleth then you were for the chop as you were clearly an Ephraimite ( who didn’t have the “sh” sound in their language) or were just unlucky enough to have a speech problem. This is supposed to have lead to about 40,000 executions! So really it was an early form of biometric identification, which sounds more advanced than some of today’s security systems.  Now I assume that with  modern-day software development, failing to pass the Shibb authentication  only results in minor annoyance and no longer leads to capital punishment, but I will be a little more careful when entering my user-name and password from now on.

 Posted by at 10:48 asides Tagged with:  Comments Off on Shibboleth: report of early stage development using biometric id.

INSPIRE and Higher Education Institutes (HEIs)

Aug 032011
 

Why is this relevant to IGIBS? I almost hear you say… well there are many reasons but for me the relevance lies in one of the outcomes for the project which is a “best practice model” for a university department’s interaction with the academic SDI. Clearly the INSPIRE regulation could have a very significant impact on that model.

  
The INSPIRE directive of 2007 seeks to establish a Community wide infrastructure for public spatial data. Relevant to IGIBS is both the likely effect of INSPIRE on HEIs and the degree of preparedness there is in the run up to the deadline for Annex III spatial data requirements in 2013.  Annex III data categories are the greatest in number (21 out of 34) and the most HE orientated. They cover a range of environmental subjects such as habitat and species distribution, atmospheric quality etc.  

 
While it is clear that almost every University will hold data that could fit into Annex III, I have found very little firm advice about what HEIs’ obligations are likely to be.   My very short and inadequate working summary  of the directive  is as follows:
To create an infrastructure in which public bodies make their policy forming and task enabling  environmental geospatial data findable, available and analysable by the wider community.

 So my first question is “will a University be classed as a public body?”  Luckily (or so I thought) there is some firm advice from DEFRA on this.  In their Guide to INSPIRE regulations 2009 they helpfully point to the Environmental Information Regulations (EIR) as a guide to defining a public body. Now in the EIR blurb they helpfully point to the Freedom of Information definitions of  a public body. Finally I tracked down a list and confirmed that Universities are likely to be a public body.  Which, after some well publicised FoI requests about climate change, we all knew anyway!  This Chinese whispered chain of advice leaves me wondering if the definition of a public body is quite as sewn up as I thought. Is it possible that for the purposes of INSPIRE HEIs won’t be classed in this way? Well the usual side step is used by me (and by DEFRA in their guide above) in that it is only the courts that make a final ruling.

 
Therefore, it is possible or even possibly likely that HEIs are public bodies for the purposes of INSPIRE. So which data might they need to make available?    I think INSPIRE regulations will be designed to only include data that are part of a body’s major role. So Natural England will need to make their data on species distribution accessible to the public and to other bodies. Whereas they (probably) won’t have to make their data on the locations of staff development sessions available; as it isn’t part of their main role as a conservation organisation.

 
This leaves me wondering what the public role of a University is. Well as I am sitting in one writing this it should be a simple question to answer. A few conversations over coffee later and there is clearly no consensus on the complete role, so my summary of suggestions will have to be….. “a mix of education, research and other less easily defined contributions to society and the economy”. Ultimately the decision about HEIs’ obligations may again be made by the courts. If I were to guess then if geospatial data is used for research or teaching then it is contributing to one of the major roles of a HEI and so could come under INSPIRE.

  
So after half a day’s work I am still looking for firm advice on what HEIs should be preparing for in the next couple of years.  Now I am sure that somewhere there are discussions going on between more astute and better informed people than myself.   What I am unrealistically hoping for is a publication from DERFA in the next couple of months that will give firm guidance to lowly short contract researchers like me, oh and Vice Chancellors. So I can finalise a report and they can direct preparations for INSPIRE which might include extending a few peoples’ contracts.

    
The best guess I can make about “ best practice”  would involve HEIs welcoming  INSPIRE, as the development of the  national SDI can only help in research and teaching and in adding value to the academic SDI. In terms of actions that should be taken, then it’s more difficult to be so certain. A minimum response to current INSPIRE regulations should involve cataloguing and metadata creation for all those poor and unfortunate orphaned geospatial datasets currently in the workhouse or on the streets, as this is not only preparation for INSPIRE but also part of  best practice.  As for any other actions well that is above my pay grade.

 Posted by at 09:45 asides Tagged with: , , , ,  1 Response »

Don’t get your Back-Up over Data Preservation

Aug 012011
 

I have been reading and  thinking about the relationship between long term spatial data preservation and the short term needs of day-to-day data security during the life of a research project. With research data being generated at faster and faster rates and the life cycles of supporting technologies getting shorter data preservation is destined to be a continual problem requiring new and smarter solutions every few years. Just dealing with new data that can be produced by Earth observation satellites at the volume of terabytes per day and may exist in several formats as is passes through complex processing stages is enough to take the issue  well into the scope of being a serious problem.  The sentiment expressed by Moss points to the aspiration of researchers for their hard won data.

“Scientists now want to keep everything, which they assume is digitally possible, in the belief that everything has value and can be retrieved and repurposed.”  Michael Moss 2008

The question is; will the technology and the resources exist to meet this aspiration?

It is very easy for a researcher in a Higher Educational institution to secure data in the short term via either their own arrangements or by using the services of a central IT department. In that way data can be backed-up to multiple locations and held on hardware that is up-to-date and covered by manufacturer’s warranties. I am sure this professional approach is found in most (or all) HE institutions. It’s still up to individuals to avail themselves of these services but there are few obstacles standing in the way. Even storage costs are falling and for a few £s per gigabyte a university department can store data in professionally managed institutional servers.  

In the hierarchy of data preservation the next levels up become harder for a researcher to arrange.  Consider keeping spatial data for third parties to discover and use for the next 5 years. Immediately there is the need for precise and comprehensive metadata. This has been addressed in several ways and the development of specific standards via UK AGMAP has given, anybody who looks, an easy lead into useful metadata creation. For this longer term data storage it may also be necessary to look outside of your home institution to ensure suitable data curation and discoverability.  But where do you put the data? It needs an accessible location that links metadata and the data object and makes them discoverable by future researchers.  The tools provided by GoGeo provide a solution. Data can be described and even lodged within this service so it becomes searchable and accessible to other researchers. 

So there is an infrastructure for this stage of the data management process but it now needs the data producer to step outside of their daily routine and to work on tasks not always considered core for a busy academic looking to their next paper.  This 5 year time horizon is also significant in that the European INSPIRE Directive will be in force   for its Annex III type data by2013. This means that many university generated geospatial data sets will need to comply with the INSPIRE standards promoting interoperability across boundaries. Possibly more difficult to achieve will be dealing with older data which will also have to meet INSPIRE standards by the next decade.

Once we look beyond the next few years and start to focus on spatial data of high quality or significance things get really interesting and much more challenging.   It’s very easy to talk of data archiving and curation as if there are standard easily accessed facilities in every library. The more I have read the more I realised that it’s a far more fluid and developing science than I appreciated. 

Who decides which data are in need of professional curation, or which data can we afford to curate?  These kinds of questions move the process beyond the researcher into the realms of professional librarians or data curators and government departments working to budgets and polices.  All this leads to a further stream of questions: Can data be given to one institution to look after? Can we guarantee that any institution will be a permanent fixture?  Will the metadata that was created during data collection still have sufficient context to be useful in 10, 50 or 100 years time? How will the increasing number of data objects be kept searchable and accessible?  With hardware life cycles only being a few years, who will ensure the passing on of data to the next technology and will that technology still support the data format? These questions just start to scratch the surface of the issues involved in designing future data curation methods and policies.

Let’s hope that the situation described in the quote below won’t be applied to the early 21st century when looking back in 20 years time.

“In terms of preserving our digital cartographic heritage, the last quarter of the 20th century has some similarities to the dark ages. In many cases, only fragments or written descriptions of the digital maps exist. In other cases, the original data have disappeared or can no longer be accessed due to changes in technical procedures and tools.”Markus Jobst 2010

It’s possible to take this timeline one stage further and start to consider which spatial data sets, that are so important to major scientific discoveries or advancements, should be considered for preservation in  the equivalent of a scientific museum that holds the essential heritage of our scientific community.

Now where did I save that first human genome I was given for safe keeping in 2003? Are well not to worry it wasent geospatial data anyway, well not unless the DNA doner had an adress?! Oh and it was a mapping project so I better fnd it…….

 Posted by at 15:18 asides Tagged with: , , , ,  Comments Off on Don’t get your Back-Up over Data Preservation

Dyfi Virtual Observatory

Jul 152011
 

As you know from previous blog entries the Dyfi Virtual Observatory (DVO) is a similar project to IGIBS and is also based at  Aberystwyth.  I had a meeting with Nicola from the DVO yesterday and we agreed to help each other out with  data sets where possible.  I thought it would be good to give a little more detail on their project so you can see how it compliments IGIBS.

The DVO is aiming to provide anyone, with an interest in the Dyfi catchment area, with a resource to process spatial data for a range of uses. Their plan is to be holders of data sets that can then be analysed and the results used privately or publicly.  To achieve this and to add processing power it is planned to make use of Cloud Computing. One example they are working on is flood risk calculation, which has obvious overlap with the  kinds of data IGIBS is looking at.

The focus of the project is more socioeconomic than IGIBS but nevertheless any extra data sets that we gain access to, that are tailored to the Dyfi area, will be of benefit to IGIBS and the end users here at IGES.

Thee `DVO web site is immenent but in the mean time you can see some impresive photos of the Dyfi area on their photostream at  flicker

 Posted by at 12:02 asides Tagged with: , , , , ,  Comments Off on Dyfi Virtual Observatory

Auntie at Ynys-hir – Springwatch

Jun 022011
 

Hope all those concerned with getting some favourable coverage for Biosffer Dyfi Biosphere are watching their televisions of an evening right now; specifically BBC2 for the next few weeks as Springwatch, that stalwart of Aunties nature programming is at Ynys-hir!

Isnt it brilliant turning on the box and seeing the intrepid trio sitting out on that decking with the evening sun on Mynydd y Llyn in the background?  I am unhappy to hear about the heron numbers going down, shocked by the cannibal Barn Owls, and flabbergasted nobody has pointed a camera at the Pied Flycatchers yet, did I miss that?  Being from Scotland, I get such a kick out of seeing these wee beauties round here – they are pretty scarce elsewhere.

 

Pied Flycatcher – photo by Rainbirder (Steve Garvie) 

Creative Commons Licence This work is licensed under a Creative Commons Attribution-ShareAlike 2.0 Generic License.

Seriously though, this is a great advert and national recognition for some of whats best about the Dyfi area, hope somebody steering Biosphere Reserve developments is taking notes.

 

 Posted by at 21:10 asides, Uncategorized Tagged with: , , , , , ,  1 Response »