Open data: a committee in retrospect

The prospect of recreating an open data committee in the Brazilian federal government prompted me to remember and tell the story of the open data committee that we created eight years ago. Please note, however, that this is not the whole story of the National Infrastructure for Open Data (INDA), or even the most important parts of it, but rather just the part that involves its committee and the issues that were discussed in it over the years.

Inspiration and motivation

Back in 2011, when we were designing the open data policy of the Brazilian federal government, one of the challenges we faced was how to ensure that citizens had a place and a say in how the policy would be run. A role model for us was the Brazilian Internet Steering Committee (CGI.br).

Established in 1995, CGI.br has representatives from multiple sectors of society: government, companies, the third sector, the science and technology community and an internet expert. While the government representatives are chosen by direct appointment, the sectorial representatives are chosen by vote from legal entities from each respective sector. The internet expert is appointed by the Ministry of Science and Technology. These rules are laid out in detail on Decree n. 4,829, from 2003.

The CGI.br was chosen as a role model mostly because it is seen as a successful example of multistakeholder good governance on technological and societal issues, as had been the case for more than a decade by then. In addition to that, the authorities at the then Secretariat for Logistics and Information Technology (SLTI) already had direct experience in participating on it, as SLTI had a seat on CGI.br, so it was an easy idea to sell to senior staff as well.

However, because we had no budget at all dedicated the nascent open data initiative, some compromises had to be made. The CGI.br has always been well funded because part of the registration fees paid to .br TLD domain name registrar by people and organizations that want to register a domain name goes to fund the Brazilian Network Information Center (NIC.br), which provides operational support for the CGI.br, as well as other related operations, such as the registrar itself (registro.br), the Regional Center for Studies on the Development of the Information Society (CETIC.br) and the regional office of the W3C in Brazil (W3C.br).

We, on the other hand, had no such luck in securing funds for the initiative. Also, we tried at the time to get the open data policy to be instated by a decree like the CGI.br is, but the political force in government of the open data movement was only enough to get an instruction norm. That’s why we had to simplify the governance structure a bit. The National Infrastructure for Open Data Steering Committee (CGINDA) was established by Instrução Normativa n. 4, in 2012, with representatives from nine government bodies, one representative from civil society, which was to be appointed by the National Secretariat for Social Articulation, and one from the academic sector, to be appointed by the Ministry of Science, Technology and Innovation (MCTI). That meant there were no elections to be held to choose the representatives, which is worse for participation when compared to CGI.br, but it also meant that we wouldn’t have to bear the cost of an election we couldn’t afford. It also meant that the committee could start working straight away and get things working fast. The operational support for running the committee would be offered by the SLTI, which would also lead it.

By then, the planning for the initiative was already being elaborated collaboratively by interested civil servants and motivated people from civil society, as can be seen registered on the planning workshop in March and further open meetings held in April, May and July, as well as the public hearing of instruction norm held in October, during the 1st National Open Data Conference. So it was only natural that the structure of the committee would have a similar composition to the participation that had already been taking place since the beginning of 2011.

First meetings

Although the CGINDA was first established in April 2012, the appointments of representatives by ministries and public bodies took some time to happen. Therefore, the first meeting took place only in October. As it was still a novelty for everyone involved, the recent Instrução Normativa n. 4 and details of the way the CGINDA would be operated were explained there. The other major topic was the first Open Data Action Plan, which was based upon the experience of the plan originally built collaboratively on the March 2011 workshop. There was a perceived necessity both to discuss the action plan extensively and to get it out quickly, so the next meeting was held in November, even though the meetings of the committee were supposed to be bi-monthly. However, the repeated requests for adjustments meant that the formal approval of the plan by the CGINDA would only happen in February 2013. That Open Data Action Plan spanned the period 2013-2014.

This first phase of the committee was marked by discussions about how to broaden civil society’s participation in it. Professor Gisele Craveiro from the University of São Paulo, the appointed representative from civil society in the name of Open Knowledge Brazil, often brought to light the necessity to stream the meetings on the internet, so that more people could watch and participate. It may seem unusual for the time and also today that discussions could happen in the open like this, but that was what people expected of us. We had just finished building Brazil’s national open data portal in a totally open and participatory manner, unlike anything that had been done around the world before. Hence, ways to improve society’s participation was one of the issues being discussed concerning CGINDA’s own bylaw, which would end up being finished and formally approved by the committee only in 2017.

Also discussed in many meetings at that time was the monitoring and the carrying out of the then current Open Data Action Plan and also the parts of Brazil’s Open Government Action Plan that concerned open data. Brazil had been one of the founders of the Open Government Partnership in 2011 and from the beginning all action plans have always featured commitments related to open data.

Getting up to full speed ahead

CGINDA’s meeting in November 2013 started a new phase in the committee. The MCTI, which had so far not participated at all, nor appointed the academic sector representative, began attending regularly. Many relevant topics were discussed then in the committee:

• the upcoming 2nd National Open Data Conference and the open data hackathon that happened during it;
• how to handle the heterogenous open data licenses of open data already present in the open data portal (which warranted later a study on the issue);
• a possible network with support from W3C.br, to build capacity around open data (which sadly never came to pass as a network, even though we did offer courses to thousands of people in public administration on subjects related to open data over the years, in partnership with Enap, the public administration school);

and more.

The Second National Open Data Conference, which received almost 300 attendees in the span of two days in November 2013, was very important and highly influential in the history of open data in Brazil. Nearly every speaker at the conference would later take an even more important role in some way in developing the open data ecosystem in the years to come. There were not only talks, but also courses, workshops on topics very much relevant to the open data agenda and an open data contest that awarded solutions and visualizations that used open data for developing civic applications.

The themes of the workshops were:

• economic sustainability of open data;
• local case studies of open data in states and municipalities;
• education and open data;
• privacy and personal data protection;
• open data licensing.

After some exposition by experts on each subject matter, workshop attendees were invited to discuss, collaborate and build a proposal to adjust the course of the open data policy. Those proposals were then brought before the next meeting of the CGINDA in January 2014. This was one more way to try to further social participation in the open data initiative.

Preparations for growth

Also in 2014 came the first proof of concept of an institution-wide Open Data Plan, shown at the CGINDA. It proposed a set of commitments that the Ministry of Planning, Budget and Management would assume to open up certain datasets, up to a specific deadline and an indication of who would be responsible for it. That kind of plan would later become instrumental in expanding the open data policy to all government bodies. At that time, the most common outcome was that some institutions with engaged individuals were ahead of the curve publishing some open data, while other institutions were lying still. So an instrument to make the publishing of new open data a regular and organized process was sorely needed.

By the end of the year 2014 and throughout 2015, progress had become slow. Despite all the advances in years prior, relatively few institutions did engage in active publication of data in the open data portal. All of it depended upon the existence of a few key influential stakeholders in each government institution to be convinced and engaged on it, as there was not much else to push these institutions forward. Despite the Access to Information Law, enacted in 2011, demanding in Article 8 that information be made available as open data, it would seldom become practice. At that time, the management of the initiative had become severely understaffed as well, at times manned by a single person (myself). That was also when the open data policy became the object of an audit from the Federal Court of Accounts (TCU).

Another problem was that some major data repositories from key institutions were still not integrated to the national open data portal, such as the National Infrastructure for Spatial Data (INDE). The technical challenge of automatically making available those spatial datasets on the open data portal was implemented in this period.

That time was also useful to evaluate and monitor the implementation of the ongoing open data plan of the Ministry of Planning. The plan would become a model for other government bodies to make their own institution-wide plan, as the Ministry of Justice would soon follow up by issuing its own open data plan.

It was also in that period that we started issuing annual reports on the progress of the open data policy. The 2014 and 2015 reports were quite simple, but they became more elaborate and detailed over the years.

Making the open data initiative scale across all federal institutions

In 2016, we finally saw the opportunity to make open data scale to every government institution at the federal level. As one of the last acts signed by president Dilma Rousseff, Decree n. 8,777/2016 established that institution-wide open data plans, such as the ones that had been running in the Ministry of Planning and the Ministry of Justice, were to become mandatory for all federal ministries and agencies (a freestyle translation of mine follows):

§ 2nd. The implementation of the Open Data Policy shall occur through the execution of an Open Data Plan within each body or entity under the direct, autarchic or foundational federal public administration, and it must include, at least the following topics:

I - the creation and maintenance of data inventories and enterprise data catalogs;

II - transparent mechanisms for prioritizing the opening of databases, which shall obey the criteria established by INDA and shall consider the data use and reuse potential by government as well as civil society;

III - a schedule of the procedures to open databases, their update and improvement;

IV - lay out clearly the roles and responsibilities of the administrative units in the body or entity of the federal public administration, related to the publishing, update, evolution and maintenance of the databases;

V - the creation of processes for engaging citizens, with the goal of facilitating and prioritizing the opening of data, explain questions in the interpretation during use and to correct problems in the data already disclosed; and

VI - other mechanisms to promote, to foster, and to effectively use the databases by society and by government.

Getting over 200 hundred public institutions to make their own plans like this in a timely manner was no easy task. Some of the key measures to make that possible were:

• capacity building:
  • take the early open data plans as examples and templates for making new ones;
  • make available a detailed, step by step manual on how to create the plan;
  • extensively offer courses on how to build the plan;
  • visit the institutions from “door to door” and offer help with removing obstacles;
• communication:
  • feature the early and well made plans as positive examples for others to follow;
  • make sure that every organization receives notice of what they must do and how they must do it;
• monitoring:
  • the Office of the Comptroller-General (CGU), which was already responsible for auditing the federal administration, got a mandate in that decree to monitor the progress of these organizations in elaborating and implementing these institution-wide open data plans.

To face this great challenge, the core team that managed the open data initiative was increased to six people. Some essential investment was also made on offering the courses that trained almost 800 civil servants with in-place workshops and almost 1,800 people in the distance course.

Besides the open data plans being the main focus during 2016 and 2017, some norms were deliberated upon and approved by the CGINDA:

• the committee’s own bylaw,
• the open data portal’s terms of use,
• a norm on how to elaborate the institution-wide open data plan.

Many others issues were discussed as well, such as: the revision of the metadata standard in light of new international standards such as DCAT by the W3C and the national application profiles used by the EU, the ongoing audit by the TCU, the open data commitments in the National Open Government Plan, participation on international conferences about open data and other international exchanges, good practices in protecting personal data before opening up data, how to treat spatial data and related metadata standards, agreements with other levels of government to federate open data portals, and more.

A recurring topic was also the placement of Brazil on international rankings about open data, mainly the Open Data Index and Open Data Barometer. I always took the time to respond the surveys and questionnaires, to look for data and other evidence that was already available and make sure it was present in the answers. One obvious deficiency has always been the difficulty in finding evidence of social and economic impact, and this has been discussed in CGINDA. Companies that use open data do not often tell in public what exactly are their data sources, so some investment in surveying those is required. Other countries in Latin America, like Mexico, had already investing in surveying private companies that use open data, carrying out case studies in GovLab’s Open Data 500. I was already bringing to the committee’s attention that the social and economic impact of open data in Brazil had to be measured and actions had to be taken to improve upon those impacts. More on the impact of open data will be the subject of another blog post of mine.

Stepping on the brakes and transition for a new management

By 2018, the prioritization of the digital transformation of services was the reason alleged by the newly appointed senior staff to decide to abstain from the responsibility of managing the initiative and to transfer it to the CGU. The body had, until then, only been responsible to monitor and audit federal institutions in their implementation of the open data policy. The CGU had also been responsible for a long time for the federal government’s transparency policy. The transparency portal had been inaugurated in 2004, even though structured data downloads came much later. In the view of those decision makers, open data promotes transparency, so efforts would be better concentrated if the leadership of the open data policy was also placed in CGU’s hands.

That is a valid train of thought. However, it can be argued that open data is not just about transparency. Individuals and civil society organizations can take open data to develop new solutions to societal problems. Companies can use open data to create new and innovative business models and also improve efficiency of existing ones. Research has shown time and time again how open data can impact even macroeconomic indicators like GDP, create jobs and reduce government spending by increasing efficiency. It would make sense to keep it at the newly created Ministry of Economy, where it was, because those were the aspects of the open data policy that needed to be most improved upon.

However, the decision was made. Personnel was allocated into other projects, and no further investment into the open data initiative or the portal was allowed until the transition was complete. I was encumbered with the task of elaborating a new decree that would effect this decision. That eventually became Decree n. 9,903/2019.

I took the opportunity to tackle a long standing problem, which was that different organizations in government would either pick different open licenses for their datasets, or would leave no notice of license at all. By defining an automatic open license for all of the federal government open data, it would no longer require managers of databases to decide about licenses on an individual basis. Any existing database rights stemming from Brazilian copyright law (Law n. 9,610/1998) that could apply to datasets that were open would then be automatically authorized for the public to use.

Article 1st. Decree n. 8,777, from May 11th 2016, is now in force with the following changes:

“Article 4th. Data made available by the federal Executive Branch and the active transparency information are free for the Public Powers and society to use.

Paragraph 1st. The free use of databases and of information made available, stemming from no. XIII from the caput of article 7th of Law 9,610, from February 19th 1998, and for which the copyright holder is the Union, are authorized in the terms set forth by article 29th of that law.

Paragraph 2nd. The federal Executive Branch is obliged to indicate the holder of copyrights belonging to third parties and the conditions for those authorized when making available databases protected by the rights defined on no. XIII from the caput of article 7th of Law n. 9,610, from 1998.” (NR)

The downsizing of the initiative meant, however, that the committee would be able to meet only three times in 2018. In early 2019, what was already slow came to a full stop when CGINDA was disbanded by Decree n. 9,759/2019, along with several other committees and commissions.

The open data plans and the new datasets did not stop, however, as Decree 8,777/2016 was still in effect requiring government institutions to do it. However, there was no more Open Data Action Plan and no more place to discuss with civil society what was important to do in open data policy at the federal level.

A new beginning for the committee?

In May this year, the CGU has placed in public consultation a new norm that would supersede Instrução Normativa n. 4 and reinstate the CGINDA with a new composition. The proposal would increase the number of civil society seats in the committee to five: two from organizations that work with fostering transparency and open data, two from research groups and one from the private sector. It also decreases the frequency of meetings to once every four months, along with a number of smaller changes. In the proposal that went under public consultation, all those five representatives from civil society would be appointed by the CGU itself. Considering the history of the open data movement and the previous discussions within the committee itself, it would be advisable that some kind of election among peers, like CGI.br has always done, be the preferred method of choosing the representatives.

There is no government-wide Open Data Plan for all of INDA currently standing, and elaborating one has already been stated as a first priority by the CGU. When doing so, there should be some kind of social participation as well. After all, as Thiago Rondon once pondered, the open data movement has an architecture of participation.