Wednesday, February 23, 2005

On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It’s normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week.

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 – 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained – the encyclopedia in its entirety was not at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and ‘watchlists’ that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly.

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called ‘write-ahead logging’ should ensure that in the event of a system crash, the database can be rebuilt from a ‘last-good’ state by replaying the edits saved in the log.

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia.

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia using Google. The articles may be viewed using Google’s cache.

Monday, December 14, 2020

On Tuesday, US-based software company Red Hat announced their plans to shift their focus away from CentOS in favour of CentOS Stream.

In the blog post Chris Wright, the Chief Technological Officer of Red Hat said Red Hat had informed the CentOS governing board that Red Hat was “shifting our investment fully from CentOS Linux to CentOS Stream”. At least five of the nine CentOS Governing Board listed on the project are Red Hat employees.

Started in 2004, CentOS has been a free-of-cost free/libre open source software which provided binary-code compatibility with Red Hat Enterprise Linux (RHEL) — Red Hat’s GNU General Public Licensed paid operating system. CentOS was absorbed into Red Hat in 2014, with Red Hat gaining the trademark rights of “CentOS”.

Red Hat also sponsors the Fedora operating system. Red Hat even gives the software engineering interns laptops with Fedora on it. Till now, software development took place on Fedora, which was later adopted in RHEL, which the Red Hat maintained and provided support for, for those customers who had RHEL subscription. CentOS would then follow RHEL to provide the same features free of cost, but without the support.

Stream was announced in September 2019, just two months after Red Hat was acquired by IBM. CentOS Stream’s development cycle had new features added to it before the features became a part of RHEL. Stream receives more frequent updates, however, it does not follow RHEL’s release cycle.

With CentOS Stream, developments from the community and the Red Hat emplyees would take place beforehand on both Fedora, and Stream as a rolling release, before those features are absorbed into RHEL. CentOS followed the release cycle of RHEL and therefore it was a stable distribution. Features available in CentOS were tried and tested by Fedora, and then RHEL maintainers. Stream, on the other hand, has features added to it before those features become a part of RHEL. That implies Stream would be ahead of RHEL’s development, containing new features which are not yet tried and tested by RHEL developers, and not be binary code-compatible with RHEL.

Writing “The future of CentOS Linux is CentOS Stream”, Wright further wrote in the announcement, CentOS Stream “provides a platform for rapid innovation at the community level but with a stable enough base to understand production dynamics.” Wright also said, “CentOS Stream isn’t a replacement for CentOS Linux; rather, it’s a natural, inevitable next step intended to fulfill the project’s goal of furthering enterprise Linux innovation.”

Since the announcement was made, many people expressed their anger on Internet Relay Chat (IRC), Reddit and CentOS project’s mailing list. CentOS 8’s End of Life (EOL) has been moved up from May 2029 to December 31, 2021, while CentOS 7 is expected to receive maintenance updates through June 2024, outliving CentOS 8.

Gregory Kurtzer, who had started the CentOS project, announced a new operating system, Rocky Linux. Rocky Linux describes itself as “a community enterprise Operating System designed to be 100% bug-for-bug compatible with Enterprise Linux”. The project said “Rocky Linux aims to function as a downstream build as CentOS had done previously, building releases after they have been added to the upstream vendor, not before.” The project maintainers have not decided a date for when the operating system will be released.

Kurtzer explained the reason to call the OS as Rocky Linux: “Thinking back to early CentOS days… My cofounder was Rocky McGaugh. He is no longer with us, so as a H/T [hat tip] to him, who never got to see the success that CentOS came to be, I introduce to you…Rocky Linux”.

Wikinews reached out to the members of the CentOS project, Pablo Greco and Rich Bowen, to discuss this move, its implications as well as the future of CentOS. Greco is a maintainer of armhfp (32-bit processors) and is a part of the CentOS’ quality assurance team. Bowen — who unlike Greco, works for Red Hat — is the community manager of the CentOS project.

Wednesday, March 1, 2006

Queensland, Australia police have charged a 57-year-old woman with the alleged contamination of the salad bar at two Sizzler restaurants.

Rat poison pellets were found in food at Sizzler restaurants at Toowong on January 20 and the Brisbane Myer Centre outlet last Sunday. Yesterday Sizzler closed salad bars nationwide, but they did not report the incidents to the Queensland Health Department until last Monday.

Queensland’s Health Minister Stephen Robertson criticised the restaurant chain for delaying reporting a possible incident of food contamination for 37 days. “I think this is a case that – a real life case – that we should perhaps learn some lessons from,” he said.

Queensland Police Inspector Bob Hytch said the woman is the same woman police asked to come forward after she told Sizzler staff she had found pellets in soup at the Myer Centre restaurant last Saturday.

The woman is expected to face Brisbane Magistrates’ Court tomorrow charged with two counts of contamination of goods and four counts of an act intending to cause grievous bodily harm.

Wednesday, July 8, 2009

Thailand’s fugitive ex-premier, Thaksin Shinawatra is in the news again today, phoning supporters in the country and appealing for no celebration of his sixtieth birthday at Sanam Luang outside the royal palace in Bangkok. This follows some red-shirted United front for Democracy against Dictatorship (UDD) supporters vowing to go ahead with the party despite Bangkok‘s Governor, Sukhumband Paribatra, saying he will deny any request.

According to Thailand’s English-language Bangkok Post, UDD leader Shinawat Haboonpad expressed determination to see the July 26 celebration go ahead, “… we will show our civil disobedience and ignore his order”.

The divisive impact of the populist Thaksin stretches back prior to him being ousted by a bloodless military coup in September 2006. As far back as 2005 figures within the Thai establishment were speaking against him; Thaksin used the courts to try and prevent dissemination of negative material, including the publication of a sermon by a respected Buddhist monk who compared him to Phra Devadhat, the Thai Buddhist equivalent of the devil. Bangkokians formed into the yellow-shirted anti-Thaksin People’s Alliance for Democracy (PAD) accusing the Prime Minister of corruption. Following the military intervention in 2006, and a groundswell of support among rural poor voters, the opposing pro-Thaksin groups formed into the UDD. Despite conviction in-absentia, Thaksin colours Thai politics, and has derailed efforts to stabilise the country’s political institutions.

This past week it has been the lead-up to the December 2008 dissolution of the pro-Thaksin People’s Power Party (PPP) government that has resurfaced. The then-Prime Minister, Somchai Wongsawat was barred from politics and his PPP dissolved by the country’s Constitutional Court following anti-Thaksin yellow shirts occupying Bangkok’s international airport and stranding as many as 300,000 tourists in the country. Now the country’s Foreign Minister, Kasit Piromya, a PAD leader, is facing pressure to step down for his part in the airport siege and blockade.

A report in Monday’s Bangkok Post indicates that Thai authorities continue to pursue Thaksin. The Interior Minister said that an attempt had been made to arrest Thaksin in Malaysia’s capital, Kuala Lumpur, but he had evaded capture and managed to return to Fiji where he remains in exile and a fugitive.

Thursday, October 4, 2007

As many as 3,200 mine workers became trapped in the in Elandsrand mine, a gold mine in South Africa about 50 miles (80 km) west of Johannesburg in Carletonville, Gauteng, after an incident on October 3. All miners have been rescued, and none of them were injured. The mine is owned by Harmony Gold Mining Corporation.

The mining company says that a lift electrical cable broke on a basket that was carrying miners, trapping thousands at least 2,200 meters (1.3 miles) below the earth’s surface. Reports from MSNBC and the Guardian Unlimited say that the shaft may have collapsed when a water or air pipe burst. The rescue operation went well with no complications, with the rescue taking just under 24 hours to complete.

“They were underground when the accident happened and they were not able to surface because an electric feeder cable that is connected to the mine lift was severed,” said a spokeswoman for the mining company, Amelia Soares. The snap was caused by a “fatigued” air pipe which burst and fell down the shaft damaging the “steelwork and electrical feeder cords,” added Soares. “They are all in good condition.”

“They are all safe. There have been no injuries or deaths,” said CEO of Harmony mining, Graham Briggs who also said that the miners were previously contacted and were given “food and water.”

Earlier reports had stated that Lesiba Seshoka, a spokesman with the National Union of Mineworkers, said that the miners have not been heard from for hours and that they could have been enduring temperatures as high as 105 degrees Fahrenheit (40 degrees Celsius).

“This is a terrible situation. The only exit is blocked, probably by a fall of ground,” said Seshoka.

Rescue workers used a mine shaft that is next to the damaged one in an attempt to reach the trapped miners, and lifted them out, 75 at a time. There was no emergency exit in the shaft which is reported to have “not been maintained for ages,” added Seshoka.

“An escape route is the most important thing and they have failed miserably. They need to ensure that the shaft is maintained. This is why we have this disaster. Our main worry is for more than 3,000 people who are underground,” said National Union of Mineworkers for S. Africa chairman, Deon Boqwana.

Current reports say that “negligence” was the initial cause for the pipe burst and that there has been a history of negligence from the mine. “We suspect negligence. Because of continuous operations there is no time to make adequate checks,” said the President of the Miners union, Senzeni Zokwana to reporters during a news conference.

The Elsrand mine is currently under development and has been since February 2001 when Harmony bought the mine. Officials state that the mine will remain closed for a minimum of six weeks, while an investigation is performed, and the mine is deemed safe for miners to continue working in it.

The mine is located in the Witwatersrand Basin of South Africa, which is said to be the location of the largest area of raw gold on the planet.

Wednesday, January 26, 2005

Santa Clara, CALIFORNIA — Computer vendor Sun Microsystems released on Tuesday the first offerings of its OpenSolaris operating system code for free, under a license which allows anyone to use or work on the code and distribute the changes they make..

The full operating system is expected to be released for download sometime in the second quarter of 2005. Following along with the system source code are 1,670 patents Sun will make available for use by open-source developers. Sun hopes to leverage greater innovation in software from the open source community, in return for allowing the code to be used without charge.

“This is not an EOL (end of life) release. It’s a supercharged, turbocharged rocket launching,” Sun CEO Scott McNealy said. “No one else has done what we have just done.”

The move follows a recent move by IBM to grant use of 500 patents freely to open source software developers. Open source developers had been becoming concerned that bureaucracy and legal issues associated with software patents could be damaging to their livelihoods and to innovation. The moves by Sun and IBM seem intended to placate this growing fear.

Dynamic Trace, or DTrace is the first OpenSolaris component to be released. DTrace is a performance-analysis tool that tells developers how to optimize their applications for maximum performance.

The code is to be released under Sun’s own licensing scheme, the Common Development and Distribution License (CDDL) Version 1.0. This license has been approved by the Open Source Initiative (OSI).

Sun’s license is based on the Mozilla Public License. The Free Software Foundation has stated on its website that Sun’s license, while free, is not compatible with the GNU Public License (GPL) used for the popular open source Linux operating system. This does not mean that Linux software and OpenSolaris software cannot be used together, but limits the extent of integration which may be achieved.

Sun had previously allowed free use of the operating system Solaris by some users, for example students. This move greatly extends the range of free uses permitted, and allows users to read and modify the source code without charge or non-disclosure agreement.

[edit]