Director's Report: DEBRIEF PART 2: Technical Details
and the Future

(Updated: Friday 5th March 5:00PM)

4 new in last 7 days

12/03/2010
Lightning Strike - Service Outage
16:35
All storage servers online. Any email sent to your account would have been accepted by our other servers during the outage and will be delivered to you shortly as the servers quickly process the backlog.
No further updates planned, service is now fully online.

16:30
2nd mailserver has been restored to working status and the failed storage server has been attended to and is now online.
Remounting into the mail cluster is taking place now. Full access should be restored within minutes.

16:15
Still no news on the fate of the unresponsive server. Disaster recovery has started (just in case) by restoring a backup of the data usually stored on it to a reserve server. No estimate on time at this stage.

16:00
Datacenter has confirmed there was a surge in power which appears to have caused damage to one of the industrial sized UPS's that keep power running to the building in the event of power failures.

15:45
Datacenter has confirmed unresponsive server is actually on but we're unable to reach it externally.
They're investigating possible causes now.

15:30
As with any outage, inbound email will have been queued and will be delivered in due course.
Still awaiting information on the one server that has gone down.

15:15
Lightning strike hit the powerlines entering the building causing the powerbreakers to overload. The UPS systems appear to have worked in that the servers themselves remained online but it knocked out portions of the datacenter network.

15:00
Our datacenter has been directly struck by lightning causing the entire power system to trip. Back on however we're still trying to determine the status of at least one of the servers which hasn't powered back up.

14:55
The network issue has caused two storage servers to need remounting. Should be fully online in a few minutes.

14:50 - 14:55
An intermittent issue with the network caused access to be restricted during this 5 minute period.


12/03/2010
Month Subscription
As announced in the Directors Report following the recent storage failures, all subscribed accounts will be receiving the additional month (30 day) subscription today. It will listed under 'My Account' -> 'Account Status / Billing' once applied.
[Update] 11:35am - All accounts have now been updated.

11/03/2010
Overnight Storage [Resolved]
One of the storage servers still hadn't mounted after last nights power breaker failure despite appearing that it had. As a result those of you with email on this server have been unable to access your email from 20:36 last night until 08:15 this morning.


We sincerely apologise for this further outage, this is the 4th week in a row we've suffered some sort of hardware failure that has affected the service and we're seriously considering all available options. Up until now this simply included new machines built with dual power supplies and feeds but discussions are now including moving to a new datacenter provider as years of trust have been wiped out with the constant stream of failures that have been outside of our control.

11/03/2010
Brief Outage [Resolved]
20:36 - 20:42
Brief outage caused by 8 servers all appearing to loose power at the same time despite being on battery/generator backups in case of failure. Currently investigating with datacenter the exact cause.


20:55
One of the storage servers isn't responding, some of you will be unable to login at the current time as a result. Should have it online in a few minutes.


20:56
Storage remounted, service fully online. Still determining original cause of the outage.

22:57
Datacenter has confirmed that a faulty power breaker tripped and had to be replaced with an on-site spare. Power was down for under 5 minutes but it took a few further minutes for the servers to boot back up. As you know we have been investigating A/B power feeds to prevent any single power fault from taking down the service.

04/03/2010
Datacenter Power Failure [Resolved]
19:37 - Power circuit failed in the datacenter, knocking out 3 servers. Service was slow as two of the servers were webservers, leaving web access restricted in speed. Client access and inbound email are both unaffected as no mail servers were on the failed circuit.


20:07 - All affected servers back online, web access stabilising across the webservers. Service may be a little slow for the next 10-15 minutes as users online are redistributed over available webservers.

03/03/2010
Temporary access issues
There was a temporary problem with access this morning between 9.45am and 10.15am due to a database server issue.


This has now been resolved.


10:25am - Updated, issue has been found and fixed.  If you have any problems, try resending your email.

03/03/2010
Thursday Emails Redelivery +Service Maintenance Notice [COMPLETED]
To those members concerned about the Thursday redelivery please be advised that this is now scheduled to happen tonight from 10pm. Other members please be aware that we are scheduling this off-peak because during this time there will be some delays on inbound mail as we process about 30,000 emails through the system. We will update this post when this has completed. 22:15 - this work is starting now. Update in approx 45 minutes. 11:05 - Mail redelivery is under way. This should hopefully complete within 1-2 hours. We will give a progress update at midnight. 12:05 - Redelivery is progressing well and should not take too much longer. Next update within 30 minutes. 00:30 - Restoration complete and full email flow restored. The held mail backlog will be cleared shortly. 01:00 - Mail backlog cleared and all operating as normal. 
01/03/2010
Attachment Issues (Resolved)
There was an issue with not being able to send attachments over the weekend.  This was caused by new attachments being unable to be written while sending.  The problem was fixed at around 7pm Sunday evening.
28/02/2010
Mail between Saturday 20th - Thursday 25th
A small portion of members will have noticed that emails are not showing up between the 20th and 25th. These have not been deleted but need to be resynced with your account following last nights server fault. This has been happening throughout the day but may take some time. Please bear with us. We will let you know when all accounts have finished syncing.





[Update]

 The bulk of the resync is complete. Some users may find messages not present from Thursday, these will be resyncing Monday morning.

26/02/2010
Temporary Access Problems [Resolved]
17:15 - Dear Members, there is a temporary mail access issue we are looking into now. Please bear with us. 17:30 - Identified the problem - working to rectify. Next update within 15 minutes. 17:45 - Subsystem is back online and we are connecting it to the front end now to restore mail access. Next update within 15 minutes. 18:00 - Service restored for about 75% of accounts. Working on the others now and will update again within 15 minutes. 18:15 - servers back online for all users. 18:20 - sorry guys - spoke too soon. One more server to go. 18:55 - sorry for delay updating this. We're having to resync the server so I'm afraid we're waiting on that to complete. Will post again within 20 minutes. 19:40 - still waiting on the sync I'm afraid. Another update in 20 mins or so. 20:20 - last step should be a final reboot. Updating again soon. 20:25 - Servers are up and we are doing final checks before pushing live. 20:30 - we have one server live for mail access but this may work very slowly. Bringing second into cluster shortly. 20:45 - second server coming up. First server coping fine and service is online. Unless there are other issues no further posts to come. Thanks for bearing with us on this. 




System Status
13th Mar @ 12:15
  • Incoming Mail
  • Web Access
  • Client Access
  • Hosting Service

 

Contents

  1. Collected Information
  2. Purpose and Use of Information
  3. Disclosure
  4. Storage
  5. Accuracy and Access
  6. Complaints Procedure
  7. Disclaimer
  8. Notification of Changes
  9. Your Comments
  10. Contact

INTRODUCTION

We understand the importance of our members' privacy and wish our users' experience to be as comfortable and enjoyable as possible. We will act responsibly in the handling of your personal data and take steps to ensure its security. The purpose of this policy is to ensure our members are always aware of the type of information we collect, how we use it, and under what circumstances, if any, we would disclose it. This policy applies to all services provided by the Aluminati Network Group Ltd.

This Policy should be read together with our Terms and Conditions and inherits any definitions defined therein.


Data protection Act

The Aluminati Network Group Ltd has submitted a voluntary notification to the Information Commissioner. Our membership number is Z8393842. All personal data is handled in accordance with the latest Data Protection act 1998.



COLLECTED INFORMATION

The information we collect is that provided by you directly via our registration form and profile interface. We also collect non-personal information for statistical analysis on the use of our websites. We also have your registered email address on record.

Any correspondence between you and us is archived for future reference.

While we keep records of any financial transactions for accounting purposes, debit / credit card details are NOT recorded, nor ever seen, by us in the case of an online payment being made. All such information is via by our 3rd party online transaction handler and we refer you to their own privacy policies.


PURPOSE AND USE OF INFORMATION

Generally the information we have is used for the normal operation and maintenance of our services. Information is used for internal purposes only such as billing and security verification of individuals. Contact details are used for internal announcements and to inform members of important information regarding the use of our web services.

Any directory system will operate on a strictly opt-in basis and will be accessible only to other members.

Use of Cookies - these are small files that reside on your system and are read by our website and webmail interface so as to enhance your experience. No personal information is stored and you may delete these cookies at any time.


DISCLOSURE

We will never give nor sell your information to 3rd parties except in the following cases:

  • You give your consent.
  • When required to do so by law.
  • In order to process and verify an online payment.

Specifically, and without loss of generality of the above statement, you can be sure we will never give your email address to any 3rd party for the purposes of advertising or promotions.


STORAGE

Your information is stored in a password protected database which is never accessible by the public nor any non-authorised employee of the Company. Authentication information, such as passwords, is encrypted.


ACCURACY AND ACCESS

We will always try to maintain our records as up-to-date and accurate and we welcome you to help us in this effort by updating your own information whenever it changes. Methods of access and modification are:

  1. Email us for a copy of your stored details and/or inform us of any changes that need to be made.
  2. Write in to request a paper print out (include a stamped addressed envelope).
  3. Access and update your profile via an online profile interface.

    We will require that you suitably identify yourself before accessing your information.

    We reserve the right to charge reasonable costs incurred by us for the supply of this information. At present there is no charge for these services.

We will refuse access for the following reasons:

  • The request does not relate to the information belonging to the person making the request.
  • Insufficient identification is produced.
  • Denial of access is required by law.
  • Providing access would pose a serious threat to the life or health of an individual or individuals.
  • The information provided for modification is reasonably believed to be false.
  • Providing access would interfere with any legal proceedings or negotiations relating to the individual.


COMPLAINTS PROCEDURE

If, for any reason, you feel the need to complain or comment about our privacy practices or policies we encourage you to contact us using the details provided below so that we may resolve the situation.


DISCLAIMER

Our web site links to and is linked from other web sites on the internet. Unless they are members of the Aluminati Network Group we accept no responsibility for their actions and we advise you to read their own privacy policies.

In the event we direct you to a secure server, for example in the case of making an online payment, it is your responsibility to check that the web site is authentic by verifying the URL as correct. You should also check to ensure the web site is secured by looking for the 'padlock' symbol or an URL beginning with "https://". In the event of any uncertainty please contact us before proceeding.

We do not accept any responsibility for any breach of privacy resulting from a member's compromised computer system or security information where the breach did not occur as a result of an action or inaction by the Aluminati Network Group. This includes, but is not limited to, the case of a password being stolen and used to access the member's profile or email account for purposes including sabotage and data theft.


NOTIFICATION OF CHANGES

Any changes to this privacy policy will be announced in the News section of the relevant websites where a link to the updated policy will be posted.


YOUR COMMENTS

Privacy is a very important issue and it is vital that we provide a policy that is reasonable, fair and responsible. If you have any comments regarding this policy then please do not hesitate to contact us using the details below.


CONTACT

Privacy Enquires
The Aluminati Network Group
4 Minton Enterprise park
Oaks Drive
Newmarket
Suffolk CB8 7YY
U.K

Email: privacy@aluminati.net
Fax: +44 (0)1638 668 890