Interesting readings

Since most of us in India can talk about little else other than corruption, do read this article by Nauro F. Campos and Ralitza Dimova on voxEU which is an interesting meta-analysis about papers which analyse the impact of corruption on growth. I have long heard about meta-analyses, but this one made me sit up and notice.

Anand Giridharadas in the New York Times on Arthur Bunder Road in Bombay.

Roger Bate and Tom Woods, in The American, point to a new dimension in India's crisis of fake medicines.

I I Sc will now use the IIT JEE as their entrance examination for the new Bachelor in Science course. Given that the IIT JEE is a well managed and difficult examination, it would make sense to have more and more schools plugging into it in order to filter their intake. But as you move away from the top .01% of the distribution, the statistical precision of the score on a very difficult exam as a measure of student capability tends to decline. The managers of the IIT JEE will need to shift towards adaptive testing, where the questions are dynamically modified based on student characteristics, in order to retain efficiency across the distribution. Once this is done, the IIT JEE would be useful for sifting through millions of students, and exert a beneficial effect of all of them facing a more demanding high-stakes examination.

Shobhana Subramanian in the Financial Express on C. B. Bhave.

A fascinating article by Nicolai Ourussoff in the New York Times about the attempt to reinvent Saudi Arabia.

Sadness about Europe by Orhan Pamuk in the New York Review of Books, and a tragic perspective on Istanbul by Claire Berlinski in City Journal.

A dystopian future for the world: a story of ageing and depopulation from Amakusa in Japan.

Liu Xiaobo's beautiful acceptance speech for the Nobel Prize for Peace. A lot of countries of the world, including India, have much to do in order to achieve freedom.

Philippines?

Tourism in Afghanistan by Damon Tabor.

Steven Johnson in the Financial Times on the future of linking to information sources on the web.

With 75% of world GDP in service, trade liberalisation in agriculture or manufacturing is not that important. The really big story is trade liberalisation in services, and there the picture is quite bad. Read this article on voxEU by Bernard Hoekman and Aaditya Matoo on how to obtain progress.

Understanding the rise in currency turnover by Michael R. King and Dagfinn Rime on voxEU.

Anders Aslund, on Project Syndicate, on the remarkable story of the global crisis as it played out in East Europe. Also see this story in The Economist on the same subject, which is a bit less optimistic. The recovery in East Europe matters for recovery in Europe and elsewhere. It also illuminates our thinking on some of the grand policy questions.

David Alexander points out how Australia is the role model for the world.

Barry Eichengreen, Daniel Gros and Ila Patnaik on the resolution of Europe's problems.

Devin Friedman in GQ on the strange world of social networking.

Discussions on 'Mythbusting: Current account deficit edition'

Many interesting comments appeared on my previous blog post, Mythbusting: Current account deficit edition, and I thought it made sense to respond to all of them in this post.

Ambarish: I don't think there has been a sudden rise in rupee trading outside India. It was always there; we weren't seeing it. As Jayanth Varma has emphasised, we used to think the NDF market was in Singapore. But the BIS data on rupee trading shows significant rupee trading at many places worldwide, not just in Singapore. Overall, the picture is roughly one with \$20 billion of onshore trading a day and another \$20 billion of offshore trading, giving an overall market size of \$40 billion a day.

One can think of many good reasons for domestic and overseas economic agents to do INR trading outside India. E.g. foreigners are, presently, not permitted to trade on the onshore currency futures. Given that gross flows across the Indian border, on the BOP, are now at \$1.3 trillion a year, it is not surprising that there is a lot of rupee trading going on. Until big changes to the capital controls take place, I believe there will be an increasing shift of INR trading away from India.

Neeraj: I agree with you that capital controls can potentially change the situation significantly. So one can think at two levels. First, for a given set of capital controls, a central bank can float as in not trading. That gives you a float, but yes, the price that comes out of this is distorted because there are capital controls. As an example, the government can have non-interference in the domestic market for DRAM chips, but the domestic price can itself be distorted through quantitative restrictions or customs duties on DRAM chips. So even though the government is not manipulating the domestic price by directly trading in the market (as it does with foodgrain or currency) the observed price is a distorted one. Then comes the second level where you have full convertibility. Once again, here the central bank could choose to trade in the market or it could choose to not trade in the market. Only when there are no capital controls + no trading by the central bank do you get to the true floating rate and the market's price.

Durga: Modulo the issue raised by Neeraj and touched upon above, I think we're a fairly flexible rate today. If INR trading globally is \$40 billion, then RBI trading of anything less than \$2 billion per day would have a negligible impact on the price. RBI has to either hit the market with very big trades (over \$2 billion a day, i.e. over \$40 billion a month, i.e. Chinese style currency manipulation) or RBI has to sit back and accept the price. Small trades are pointless, and actually make you wonder what the strategy there is.

That said, the rupee is still a small currency. India is a GDP of only \$1.25 trillion and there are a lot of restrictions on cross-border commerce. So there is a long way to go before the INR becomes a serious international currency. It does not, hence, surprise me to see that the spreads on the INR are much worse than those seen for the big international currencies.

However, what I talked about in the blog post -- that when a central bank stops trading on the currency market, the CAD = capital flows -- is not an equilibrium condition. It is an accounting
identity. It requires nothing about market microstructure on the currency market, or about the capital controls, in order to hold. As long as RBI trading on the currency market is zero, CAD will be exactly equal to capital flows.

Finally, Anonymous, you ask: Is there a point where the CAD becomes so big that it becomes dangerous? We should think in two parts.

First, in a place like the US, there has been a lot of concern that the imbalance (= the very large CAD) is too big in the sense that under reasonable assumptions, the US is not going to be able to service all the capital coming into the country. After all, all the equity / debt capital that comes into (say) India today inexorably requires that at future dates, dividends and coupon payments and debt repayment have to happen in dollars, which will require purchases of foreign exchange by residents. In order to service the borrowing of the US today, substantial exports growth will be required, which is unlikely. Hence, when this borrowing of today is to be repaid in the future, a huge dollar depreciation will have to take place.

As long as there is an environment of high growth in exports of goods and services, there is no problem. If, hypothetically, you see a country with a big CAD but you also have a WEAK pace of exports growth then you know that at future dates, there will be pressure on the currency which will give sharp depreciations. Odds are, the financial system will see that and these fears will translate into a depreciation right away! Conversely, if you see capital inflows going into investments which will bolster growth of exports of goods and services, then you feel comfortable that there is no problem.

I believe that's a fair description of the present Indian situation. Over the last 15 years, the gross inflows on the current account into India (which can be roughly interpreted as the total revenues from exporting goods and services) grew by 8.1 times, from \$42 billion in 1994-95 to \$343 billion in 2009-10. This was an average annual growth rate of 15%. This is a huge pace of growth, and gives me confidence that the CAD coming in today will be serviced tomorrow without large currency depreciation. If, hypothetically, you disagree with my optimism about future growth in exports of goods and services, then you would think that this large CAD today increases the odds of INR depreciation in the future, and you would go short the rupee.

For smaller emerging markets, there is a risk of sudden changes in international financing conditions, which is rooted in the lack of information in the hands of foreign investors about the country. Then a large CAD could mean that if something goes wrong and a lot of capital leaves the country, then it could yield a large currency depreciation. I believe this is less and less an issue for the large emerging markets like India, where problems of asymmetric information and lack-of-attention in the global community are not a problem.

Mythbusting: Current account deficit edition

The question

In recent months, the current account deficit has risen. The latest data shows:

Sep 2009	-3.03
Dec 2009	-3.64
Mar 2010	-3.68
Jun 2010	-3.84

This has started making many people worried. Is such a `large' current account deficit a cause for concern?

The right answer

How long should a man's legs be? Long enough to touch the ground.

The old intuition

Under a fixed exchange rate, where the central bank holds the rate fixed by trading on the market:

• Net capital inflow is an autonomous variable
  
• All the capital that comes into the country is bought by the central bank (and vice versa), and this has consequences for sterilisation or monetary distortions.
  
• You can then ask yourself whether the amount of capital coming into the country is "too much" or "too little".

The new intuition

But all this changes under a floating exchange rate!

As the graph above shows, RBI's trading on the currency market has been at near-zero values in recent months: we have something that is essentially a floating exchange rate. The rupee is now a fairly big market, and small scale trading by RBI has zero impact on the price: i.e. what we're seeing is a true market price. Under a floating exchange rate:

• Net capital inflows = Current account deficit, as an accounting identity
  
• If there is a sudden increase in capital inflows, this yields a rupee appreciation, which tends to increase the current account deficit. Conversely, if there is a sudden capital outflow, this yields a rupee depreciation, which tends to decrease the current account deficit. Through this, there are constant equilibriating forces which bring the two together.

With a floating exchange rate, you curiously look at the current account deficit and wonder that if there is some sudden international crisis (e.g. Lehman's death) whether there would be a short-run dislocation. For the rest, there is no policy involvement in either the current account deficit or in net capital inflows, both of which are purely market phenomena.

A new angle

In the very short run (e.g. a day), changes in the exchange rate can have little impact upon imports or exports. So if \$10 billion suddenly leaves the country in a day, when the rupee depreciates, there can't be a response from import or exports immediately. The only response that can come about immediately is: from capital flows.

When \$10 billion leaves the country, the rupee depreciates, and some investors think that they will score some nice returns by buying short-dated rupee securities. They step in in the breach, thus yielding an equilibrium.

So I will conjecture: A country that has capital controls against short-dated debt flows will have more volatility on the currency market.

Also see

Viewing the current account deficit as a capital inflow by Matthew Higgins and Thomas Klitgaard, FRBNY, December 1998.

Previous editions of `Mythbusting'

Mythbusting: Reserves edition, 18 October 2008.

Talk on US financial regulatory reform, by Viral Acharya

Viral Acharya will do a talk Recent developments in financial regulatory policy in the United States: Review and Critique at NIPFP (1st floor conference room) at 4:30 PM on Monday the 20th of December. All are invited. The talk will be followed by snacks on the lawns of NIPFP.

This will draw upon the work of many scholars at the NYU Stern School of Business, which has given two books: Restoring financial stability: How to repair a failed system (Viral V. Acharya, Matthew Richardson, 2009) and Regulating Wall Street: The Dodd-Frank Act and the new architecture of global finance (Viral V. Acharya, Thomas F. Cooley, Matthew P. Richardson, Ingo Walter, 2010).

Appropriate regulatory structure for development

A. D. Shroff Annual Public Lecture, by C. B. Bhave.

It is a great honour to be invited to deliver the A. D. Shroff Annual Public Lecture. Mr. A. D. Shroff was an outstanding financial thinker and a practioner who took great interest in organisational and ideological issues. He was known to express his views in a candid manner and without any fear of the consequences of such expression. Regulators have a reputation of not speaking much and if they do speak then not saying much. I will try to strike a balance between Mr. Shroff's forthrightness and regulatory reticence.

Costs and benefits of regulation

The world has gone through very troubled times in the last three years. Unbridled growth and development in the financial markets is no longer an accepted article of faith. Deregulation in developed markets resulted in excessive leverage being built by large institutions, and financial innovation being used more to hide risk than create real value. This inevitably led to a crisis and the cost of repair is being borne by the tax payer and the economies in general.

Those who are bearing the costs are, in a substantial measure, not those who reaped the benefits of unchecked growth. In the event, there is no support for development without regulation. For orderly development, regulation is a sine qua non. Notwithstanding the fact that regulation is a must for orderly development, we still need to enquire and debate what constitutes an appropriate regulatory structure. We need to debate issues around this especially in the Indian context.

At the very basic level, regulation means restraint and restraint is a hindrance. Thus any business subject to regulation does pay a price whether the regulation is voluntary or imposed. The question is not whether regulation will come in the way of development but whether the price we pay by accepting regulation is worthwhile or not.

Three kinds of regulation

If we look at various sources of regulation one can roughly say that there are three reasons why business entities agree to regulate their behaviour even though it does make them pay a price for such regulation or restraint:

• The first source of regulation arises from the fact that the commercial entity interacts with the outside world, suppliers, customers, financers, shareholders and so on. There are certain norms by which the entity decides to bind itself irrespective of whether there are formal rules and regulations or deterrent punishment for deviation from norms exists or not. No trader can repeatedly violate his contract even if oral, with either his customer or with his supplier. It will simply render his business impossible. One can call this self regulation at its most basic level with the source of discipline being the market place. The market place simply does not deal with you if your behaviour is substantially out of line with basic norms and we don't need the force of law here to enforce such norms.
• As a second source of demand for regulation one can look at situations where entities engaged in a particular business activity may decide to come together and conclude that certain norms of behaviour are not adequately discouraged if the entire thing is left to the individual entities. Yet, the group feels that such norms need to be in place for the overall development of their business. Since such voluntary groupings of entities do not enjoy the force of law they may decide that any behaviour against the agreed rules of behaviour will be punished by making the concerned entity lose the membership of that group. Trade Guilds, clubs, the early form of stock exchanges are examples of this. This form of regulation is commonly known as self regulation. This self regulation is not regulation of activities by the entity by itself but is the regulation of the entity by a common interest group of which that entity has agreed to be a member. For such a grouping to succeed, individual members must be able to see the benefits of membership. The price of being expelled from membership should be high enough to ensure behaviour as per the commonly agreed norms by the group itself. Our experience in India has not been entirely satisfactory in this area. Nevertheless, we need to continue our efforts at establishing credible self-regulation.
• That brings us to the third category of regulation which is regulation enforced by law. The argument in such cases appears to be that the activity of entities in a particular area of operation affects the lives of more than just the member entities. In other words the society has a stake in ensuring that the entities conduct their operation in a manner that is acceptable not just to those entities but to the society at large as well. The discontentment with financial meltdown is very aptly captured by the expression `privatisation of gains and nationalisation of losses'. This sentiment is also a reflection of the fact that there are stakeholders outside the universe of finance who suffer if finance is not regulated.

The interplay between self regulation and regulation by the authority of law has been a subject of interesting discussion not only in the area of capital markets but in other fields as well. Self regulation is generally considered desirable since it is made by the entities themselves and therefore,it is considered more business friendly. Equally there are arguments that there are not sufficient incentives in self regulation to put the interest of other stakeholders before the interests of the participating entities. In addition self regulation lacks the ability to enforce its rules beyond depriving the member concerned the membership of the group. If a significant group decides to violate norms the self regulatory structure can become unsustainable and only the backing of law can sustain such activity.

In different jurisdictions, efforts have been made to make the deterrent actions of self regulatory organisations stronger by granting such organisations `recognition'. However, difficulties arise if more than one organisation wants to be recognised as a self regulatory organisation for entities in the same area or business. In other words if the entities split and form multiple organisations, all of which seek recognition as self-regulatory organisations, the situation is not amenable to an easy resolution. Notwithstanding the various forms of self regulatory organisations and the different degrees of strength and their deterrent actions, it is commonly accepted around the world that self regulation alone is not sufficient and an apex regulatory body is necessary.

The functions of the regulator

Regulation with the backing of legislation is administered either by the Government itself or their autonomous statutory regulatory organisations. While the model of Government being a regulator itself has been tried in the past,the modern consensus is to have independent and autonomous statutory regulatory bodies. In the wake of the reforms undertaken by the Government in 1991, SEBI legislation was passed by the Parliament in April 1992. SEBI has been created as an independent statutory body.

What are regulators expected to do? Regulators set rules for conduct of market entities, the manner of conducting business, and even the tariff to be charged in certain cases. Regulators may also lay down norms for entry as well as continuity of business for entities. It is thus apparent that regulators can enjoy powers in the area of rule making for entry / exit regulation, conduct regulation, tariff regulation, and risk containment regulation.

Regulators not only set rules but are also required to keep an eye on the compliance of these rules. They therefore, end up setting up an elaborate mechanism for ensuring compliance. If despite this, the rules are breached then the regulators are charged with the duty of carrying out necessary investigation and enforcing these rules by adjudication.

The question of autonomy of the regulator

The list of responsibilities is fairly onerous and since the regulators combine in themselves the roles of rule making (legislative role), administration of rules and investigation if breach of rules occur (administrative function) and adjudication (judicial function), it is necessary to pay careful attention to the governance issues of regulators. It is an accepted principle that regulators need to be autonomous in discharging the duties laid down by law. A regulator, subordinate to or dependent on the executive wing of the Government will not be in a position to do proper justice to its duties.

Autonomy is not only a matter of creating appropriate structures and legal provisions but also a matter of perception. Regulatory structures in India are in different stages of evolution and therefore the thinking on autonomy and the perception of autonomy has not yet fully crystallised.

The Reserve Bank of India as a regulator has been in existence for more than 75 years and therefore, the relationship between the executive branch of the Government and the RBI is far more evolved compared to the relationship of regulators which are of more recent origin. SEBI is in its 19th year and stands somewhere in the middle of regulatory evolution: it is more evolved compared to the regulators that have been set up in this century but has lesser history when compared to the Central Bank.

The first Chairman of statutory SEBI, Mr. G. V. Ramakrishna, once famously remarked in the early days that brokers of BSE should know that the route from Dalal Street (BSE) to Mittal Court (the location of the SEBI head office, then) is not via the North Block (Finance Ministry, Delhi). The brokers at that time had not got used to the idea of a regulatory body having been formed which would independently set regulations. Capital market regulation was part of the Ministry of Finance functions till the formation of SEBI. They therefore had a tendency to run to the Government for every little problem.

The tension between the executive branch of the Government and the regulatory bodies is not a phenomenon only during the early stages of regulation nor is it peculiar to India alone. Both the regulators and the executive need to nurture this relationship in a manner that reinforces regulatory autonomy. It is not easy for the executive to deal with this especially when the very powers that were exercised by the executive are transferred to the regulator. It is imperative in this context to make sure that there are adequate supportive provisions in law and the rules to support the autonomous character of the institutions.

To maintain the autonomous character of the institutions and its independence from the executive one needs to start at the process of the appointment and the terms of removal of the Members of the regulatory apparatus. Interestingly, the framers of the Indian Constitution saw the importance of this aspect in institutions such as the Election Commission, the Higher Judiciary namely High Courts and Supreme Courts and the Comptroller and Auditor General of India. The Constitution makers were very careful in providing for the conditions for removal of persons at the helm of these bodies even while recognising that the appointments will be made by the executive. These autonomous institutions have served India well. The prolonged tension between the Election Commission and the other organs of the Government is an example of how constitutional protection delivered a powerful and autonomous Election Commission which admirably served the cause of democracy.

The regulators do not enjoy protection in terms of the conditions under which their services can be dispensed with by the executive. In fact the regulators are at the other end of the spectrum in terms of provisions for their removal. In SEBI, the Members and the Chairman are appointed for a tenure of certain number of years or until further orders whichever is earlier.

A tradition has been established that regulators are not removed from their jobs as easily as the functionaries in the executive itself. There is no known example of the executive having resorted to the clause `until further orders whichever is earlier' to remove the functionaries of the regulatory organisations. Whether it is sufficient to rely on tradition or whether we need a better legal mechanism with checks and balances needs to be debated, so that this important aspect of governance is not ignored.

A vital component of autonomy is financial autonomy. In case of SEBI and some other regulators such as IRDA this autonomy was built into the legislation by way of providing that such authorities will establish a separate fund into which the fees paid by the market intermediaries will be credited. Such funds are to be used by the authorities for discharging the functions entrusted to them by law.

Currently there is a line of thought - as you must have all read in the media - that the regulatory authorities should not be allowed to have funds of their own but these funds should be merged with the Consolidated Fund of India. If the Government finally accepts this line of thinking, substantial damage will be done to the autonomy of regulatory institutions. If the regulators have to depend on the executive for release of funds the question of independent behaviour by the regulators would be jeopardised. It is necessary to carefully consider the pros and cons of taking away financial autonomy from regulators.

The function of investigation in case of breach of rules is an area that hinges in a vital manner on autonomy from the executive wing. Regulators by the definition of their responsibility have investigative wings. This function has come under increasing judicial scrutiny and the movement of the last 15 to 20 years has been to free the investigation function from the possibilities of influence by the executive.

The CBI is a case in point. Under the direction of the Supreme Court the supervision of this institution is with the Chief Vigilance Commission which in itself is an independent statutory authority. I would therefore, argue that regulatory autonomy vis a vis the executive wing of the Government is not only necessary but is essential.

The question of accountability

Any governance structure based on autonomy must also look into the question of accountability. Since regulators have multiple roles, part legislative, part administrative and part adjudicatory, the accountability in the three areas is handled in different ways. Regulators are creatures of law and the ultimate supervisory authority of the Parliament to assess whether the regulators are discharging the functions assigned to them is supreme.

The Comptroller and Auditor General of India is empowered under the regulatory provisions to audit accounts of the regulators and submit reports to the Parliament to help the legislative in its assessment. In addition the regulators are required to prepare an annual report on their activities and lay it on the table of both Houses of Parliament.

The adjudicatory function of the regulators has been treated differently and by its nature has to be a subject matter of supervision by judicial bodies. A mechanism in the form of Securities Appellate Tribunal headed by a retired High Court Judge and an appeal provision to the Supreme Court of India forms an integral part of SEBI legislation.

The rule making powers of SEBI are supervised by the Parliament in order to ensure that the rule making is confined to the powers granted by the Parliament to the regulators. If a regulator exercises power beyond the permissible limit of legislation, the rules can also be challenged in the courts of law.

In the rule making function the regulators do interact with the executive branch of the Government. The executive wing of the Government will have legitimate imputs into the rule making process and a fine balance is required between the need for autonomy and the need for harmonisation. This is achieved through the presence of Government representatives in the Board of SEBI.

Conclusion

In conclusion, it is quite clear that attempts at unregulated development not only in a particular sector but even in small sub-sections of sectors have failed. The failure is mainly because such development ultimately leads to crisis. The cost of resolving such crisis is high and the burden of the cost is borne not just by those who benefited from the development but a large portion is borne by those who were not part of the recipients of the benefits. Clearly the collateral damage is very high.

The question is, therefore, not so much as to whether development and regulation are in conflict as the quality of regulation that will enable us to find a balance between the needs of development and the need to keep the risk-reward relationship appropriate. It is necessary to carefully think and design proper regulatory structures, ensure regulatory autonomy and make sure that there are checks and balances in the system to address the concerns of accountability as well.

Thank you.

Emerging markets finance conference at IGIDR

link

A puzzling data revision

Ordinarily, official statistics get revised because at first, provisional estimates are released, and when the full data filings come in, then improved estimates are put out.

In the case of RBI's data about RBI's trading on the currency market, such data revisions should ordinarily not arise.

But yesterday, data released by RBI modified the previous information that had been put out about RBI's trading on the currency market. Earlier, trading in June had been claimed to be 0. Now it shows purchase of $370 million and sale of $260 million. Earlier, trading in September had been claimed to be 0. Now it shows a purchase of $260 million. I wonder why this data revision took place.

The newest data - for October - shows a purchase of $450 million on the spot and $450 million on the forwards. At a time when rupee trading is estimated at above $40 billion a day (worldwide), it is hard to see how such a small scale of trading can generate a significant impact upon the price; so I wonder what is going on in terms of the rationale and the thought process.

Interesting readings

One of the big impediments to India's integration into the world economy is xenophobic visa rules. There is some progress in the pipeline: visa on arrival has been working from Jan 2010 onwards for visitors from Finland, Japan, Luxembourg, New Zealand and Singapore. A nice touch here is that India did not get stuck on issues of reciprocity; this is unilateral liberalisation.

Watch this talk by Steve Coll.

Mature treatments of the Niira Radia wiretaps : Sail Tripathi in the Mint, Pratap Bhanu Mehta in the Indian Express.

Anil Padmanabhan in Mint on the question of corruption, and Sevanti Ninan on the media response to the tapes.

In search of America's liberty and India's dharma by Gurcharan Das in the Times of India.

A rumination by Vikram Doctor, on the need to shift focus in Bombay from the West to the East.

Sam Geall on the problems of Chinese science. Some of these problems are found in India also.

With corruption scandals galore, what India needs most is competent and clean government. SEBI continues to soldier on: see the recent order on bond issues by Sahara. Or if you don't have appetite for the full text, here is a precis by V. Umakanth. Everyone interested in Indian finance should read a few orders of Bhave's SEBI every year: they give you fresh insights into how the interplay between law and regulation works.

Tamal Bandyopadhyay in Mint with his sense about the extent of corruption in Indian banking.

How do foreign capital flows behave around elections, on voxEU by Emmanuel Frot and Javier Santiso.

Currency warriors should consider India by Sebastian Mallaby in the Financial Times.

A. K. Bhattacharya writes in the Business Standard about fresh thinking on Indian Railways from an unexpected source.

Huang Yiping on voxEU has a story from China which is similar to what we often see in India: the use of microeconomic tools to go after macroeconomic problems.

In the footsteps of Gandhi, Mandela and Havel, by Ma Jian, on Project Syndicate. Unveiling hidden China by Christian Caryl in the New York Review of Books.

Good-bye to Dubai by Joshua Hammer in the New York Review of Books.

Robert Messenger looks back at Dien Bien Phu.

Richard Boudreaux in the Wall Street Journal about Russia's Parliament accepting Stalin's responsibility for the Katyn massacre.

Kenneth Rogoff on the Euro.

A tale from the frontiers of public administration. The Australian government has announced a competition to forecast the behaviour of traffic on Sydney's M4 freeway. This illustrates three themes. The first is that of better living through science: the attempt at using statistical analysis to shape public administration. The second is the unique value of public domain databases. The third is the importance of harnessing brainpower out there in innovative ways: through openness of data and through the competition.

Trailhead by E. O. Wilson. As I read it, I was astonished at the way in which knowledge gleaned from hundreds of research papers has been stitched into a compelling story.

A club of 19

What binds this club of 19 countries: China, Russia, Kazakhstan, Colombia, Tunisia, Saudi Arabia, Pakistan, Serbia, Iraq, Iran, Vietnam, Afghanistan, Venezuela, the Philippines, Egypt, Sudan, Ukraine, Cuba and Morocco? Answer. Am I glad India is not in this club!

Alternative stock market indexes

I saw this interesting article about the mind-share of Nifty as opposed to the BSE Sensex. It is by Samie Modak and Muthukumar K. in the Financial Express.

The NSE data for June 2010 shows that Nifty futures have peaked at Rs.0.36 trillion of notional turnover in a day (27 Jan 2010) and Nifty options have peaked at Rs.0.89 trillion of notional turnover in a day (24 June 2010). Nifty has shaped up as one of the big contracts by world standards. It is interesting to go back and read the original paper. Those were interesting times. Looking back, it seems obvious that Nifty would dominate the derivatives market, but at the time, the outcome was far from clear.

This made me look at data on risk and reward of the alternative indexes. I start from the first data for Nifty Junior, which takes me back to 21 February 1997, thus giving data for 13.7 years.

	Mean	Volatility	Ratio
Nifty	12.99	26.37	0.4926
BSE Sensex	12.68	26.92	0.4711
Nifty Jr.	18.16	32.38	0.5608
CMIE Cospi	17.40	27.23	0.6391

Nifty and the BSE Sensex are a lot like each other.

The real surprise is Nifty Junior: Merely moving down from rank 1-50 to ranks 51-100 has given an enormous juice in the return and in the reward-to-risk ratio. But the volatility of Nifty Junior is also higher.

The CMIE Cospi index has roughly 2800 stocks today, and represents the broad market. It includes the Nifty Junior stocks and a host of other smaller stocks. But unfortunately, these numbers are not comprabale with the other three in that it includes dividends while the other three do not. With this combination of high diversification (giving a low volatility), small-cap stocks (which helps returns) and inclusion of dividends (which helps returns), it is not surprising that it scores the best reward-to-risk ratio.

In my mind, most of the claims of out-performance by active managers in India are purely about being invested in the non-Nifty space. Nifty Junior ETFs are easily accessible and I get surprised that more people aren't putting this into their investment strategy.

A more efficient piece of code

CMIE's firm databases use a fine-grained product code to identify each product. Each firm is also allocated to a product code based on its predominant activities. I like to reconstruct a coarse classification out of this that suits my tastes. I do this using this R function:

cmie.14.industries <- function(s) {
  values.8 <- c("Food","Textiles",
                 "Chemicals","NonMetalMin",
                "Metals","Machinery",
                 "TransportEq","MiscManuf",
                "Diversified","Serv.IT")
  names(values.8) <- c("01010101", "01010102",
                       "01010103", "01010104",
                       "01010105", "01010106",
                       "01010107", "01010108",
                       "01010109","01010408")
  values.6 <- c("Serv.Construction","Serv.Other",
                 "Mining","Electricity")
  names(values.6) <- c("010106","010104","010102",
                       "010103")

  if (is.na(s)) {return(NA)}

  leading8 <- substr(s, 1, 8)
  attempt <- values.8[leading8]
  if (!is.na(attempt)) {return(attempt)}

  leading6 <- substr(s, 1, 6)
  attempt <- values.6[leading6]
  if (!is.na(attempt)) {return(attempt)}

  leading4 <- substr(s, 1, 4)
  if (leading4 == "0102") {return("Serv.Finance")}

  return("MISTAKE")
}

This maps each firm into one of 14 coarse categories. Here are some examples of this in action:

> cmie.14.industries("0102090000000000")
"Serv.Finance"
> cmie.14.industries("0101041502000000")
"Serv.Other" 
> cmie.14.industries("0101010601010000")
"Machinery"

So in short, the function cmie.14.industries() maps a string like "0101010601010000" into a set of 14 broad industry names such as "Machinery".

Faced with a file with roughly 48,000 firm-years, at first blush, it seems that this function has to be run 48,000 times. For a given firm, this classification could change over time, so it isn't just a matter of doing this once for each firm. Here is one simple way to do it:

badway <- function(task) {
  result <- rep("", length(task))
  for (i in 1:length(task)) {
    result[i] <- cmie.14.industries(task[i])
  }
  result
}

This is just a loop that runs over everything in the supplied vector and calls cmie.14.industries() for each element. The only concession to efficiency is that the empty vector `result' is allocated ahead of time.

This proves to be quite slow. None of the standard R vectorisation ideas offer much relief.

The key idea for obtaining a leap in performance was that while I had to run through 48,000 firm-years, the industry codes actually attain only a modest list of possibilities. This makes possible a table lookup:

goodway <- function(task) {
  possibilities <- unique(task)
  values <- rep("", length(possibilities))
  for (i in 1:length(possibilities)) {
    values[i] <- cmie.14.industries(possibilities[i])
  }
  names(values) <- possibilities
  values[task]
}

For a problem of size 1000, this works out to be 13.5 times faster:

> load("task.rda")
> length(task)
[1] 1000
> system.time(res1 <- badway(task))
   user  system elapsed 
  0.030   0.000   0.031 
> system.time(res2 <- goodway(task))
   user  system elapsed 
  0.002   0.000   0.002 

This is just a demo with a 1000-sized task. In my production situation, the performance difference is even greater, since badway() calls cmie.14.industries() 48,000 times while goodway() only calls it a few hundred times.