Updating an ASP .NET web service without disconnecting users

[Reezer]

The project I'm currently working on uses an ASP .NET web service to synchronise some data periodically from a set of client machines.

At the moment when we want to update the web service we have to stop the service, kill the iss process, update the dlls and aspxs then restart iss and the service.

Microsoft's site says that you can update on the fly, and this is true for the aspxs (if we change one using notepad the change will be available to the next client to connect) but any dlls we use appear to be cached somewhere.

Does anyone know of a way to update the entire service without disconnecting all the clients?

We plan to get two versions running side by side and change the client to automatically switch between them when disconnected which will make updates easier but since ASP .NET is supposed to handle this kind of thing we were hoping to find a better solution...

Anyone got any ideas how to achieve this?

 

[Reezer]

Send an email to all users "There will be a short period of inactivity"
Replace old with new..

ctrl+r - iisreset

Done.

Tis the best way in all honesty.. if you go fishing for old DLL's you are bound to miss one.

Unless IIS has some sort of refresh command.

EDIT: after google, it appears it doesn't

We can't email the users, we don't know who they are... we wrote and now host/maintain the software for a finance company who have installed it on hundreds of brokers/dealers systems. We would have to get a dpa waiver from each user to use the the contact info the system holds (it probably wouldn't be a problem but isn't being done at the moment.) We were planning some kind of message to users to inform them of the pending disconnect but that relies on them doing something to cause the synchronisation before the disconnect happens.

I've read about the dynamic update features of asp.net but it seems pretty crap if you can't update the dlls your ASP .NET references when you update the service code. Finding the dlls isn't a problem, they are all in the App_Code directory, we replace these every time we do a build, the problem is getting IIS to refetch them when it recompiles the service pages.

I suppose in the long term we could put all the source for our dlls in the service solution and let them all be compiled on the fly but the idea was the synchronisation dll could be used from a web service or embedded in a server application.

I suppose having a redundant service would be a decent solution because it would handle the case where one server goes down unexpectedly as well as the updating problem but it means another server will be sitting there doing nothing 99% of the time. I'll speak to our techy and see if we have some crappy old machine that could be used to field requests for a couple of minutes every couple of weeks and do something non-critical the rest of the time...

 

[Reezer]

The client refused any redundat servers, we buy them on a per-client, per-app basis and we quoted them for two redundant sync servers and one redundant web management server/terminal but they refused the extra cost. Their brokers are motor equipment traders. Our system basically accepts their deals offline then syncs them up to the finance companies server for appoval and payout.

If it had been down to me we'd have the redundancies but having one of our servers running their redundancy that they aren't willing to pay for probably wouldn't go down well with our money man. Pfft, if it had been down to me we'd have done some proper specing, testing and have had a proper user signoff so that these panic changes were handled on a case by case basis and probably chargable rather than us doing everything they come up with when they click their fingers (which causes more problems because they won't wait for internal or user tests.) I think we should put our food down about a variety of things but the finance company in question is ~ 3rd biggest in europe and we're a small company not willing to upset them. Short term pain would mean long term gain rather than our current short term pain followed by months of long term pain because nothing is properly tested.

Bit off topic but it's a very big part of the problems we're having now...

 

[Reezer]

Sorry to say it dude, but I honestly cannot see your company keeping that contract :/

Well they have launched it uk wide to a resounding success and now want to go europe-wide after we work out all the kinks. They have already put nearly 30 million quids worth of business through it so I can't see them backing out somehow

It's nothing to do with us really, we provide redundant servers for our other big client, 3 layers of redundancy in fact. Should the lack of redundancy become a problem then we have a bit of paper saying "we don't want the redundant servers against your best advice". They will probably realise the usefulness of them once the number of users goes up 50 fold and they start asking for new features.

We have two test systems running copies of their system and we only need one of them at the moment (no new development going on at the moment) so we are going to try and set up redundancy on that and say "look how useful this is, buy another server you cheapskates". I would imagine once they go europe-wide they will want to host some servers in their HQ "back in the motherland" in which case it will be their problem when one goes down or catches fire or whatever.

Either way, the ASP .NET dll updating only works if the ASP .NET dynamic compile service compiles the code. There is actually another way we can separate our client/server stuff, the only real commonality is the "Message" classes which are changed extremely rarely. I may try to refactor it so that the server side stuff is entirely within the webservice project and the client stuff is in a different dll which should allow for they dynamic updating. The redundancy would be a very good idea from an uptime point of view and would also solve this issue but if it's out of our hands unless our company fancies forking out a load on new hardware and software licences out of the goodness of our hearts.

Sorry If I'm not following here, but could you not do it at 3 in the morning?

Technically we could but it would mean someone coming in 3am whenever an update is needed which would require them getting security clearance have the "master" keys and access the servers with no one else present. If this were the only solution we would do it, I'm just exploring some other avenues which are hopefully simpler and more developer friendly

There are a bunch of solutions but the ASP .NET dynamic updates was specifically designed to solve this problem in "the best" way, it's just a shame it doesn't work like it should

 

[Reezer]

Client: "Who authorised this change?"
You (company): "Er.. we did?"
Client: "You're fired!"
(Judging from your lack of knowing who your clients are!)

or

Client: "Quick, the system is down and we haven't paid ourselves our bonuses yet! Help!"
You: "Er, that'll be some time next week once the replacement hardware has arrived and we have it up and running."
Client: "Just what the hell did we give you £30mill for?!"
You: "Er.."
Client: "You're fired!"

If anything, this is dealing with financials and doesn't have a redundant system.. Sarbanes & Oxely Auditors would have a field day and terminate the contract for you..

Think I've given the wrong impression - the 30 mil is the amount of money including profit and costs that they have put through the system (i.e. lend 100k, get 110k back, 110k is the amount "put through system".)

We know exactly who our client is, what we don't know (or rather, the information we aren't allowed to use) is which of their customers they have installed our client software on.

Also, "Er.. we did?" becomes, "Er... You did, and here is your signature and an email from your superior to prove it". They approve all updates, client and server - we do internal testing, followed by them doing user acceptance testing and then finally getting a signature from the guy in charge over there and an email from the UK MD to acknowledge that he knows the update is going on.

We also have a service level agreement for technical services, development, support and a bunch or other stuff we do for them which clearly states there is no redundancy. Technically the loss of the server isn't the end of the world anyway - all the deals are still stored on the original dealers machines so the updates can be resynced as soon as the server (or a replacement) is made available - the worst case scenario is they have to wait 12 hrs (max) more for their deal to be underwritten; the client say they will try to underwrite in 24 hours and usually do it within 2 so we've still got 10 hours to spare if it all goes pear-shaped. It's not like we don't have other racks that *could* be used in an emergency, it's just they are ours for internal use and aren't paid for by the client.

We are not at any risk of getting shafted by the client - we've get very tight contracts and have fulfilled all of our obligations and provided more support than contracted. If it came down to it they would have to choose, keep paying us or loose the use of the software - they have been granted limited rights to use it, we still own the intellectual property and copy rights. For us, the worst possible case is we stop getting their money but still have a framework that we can sell to other finance companies whereas for them, they get a dump of the database and no way to use it nor any software to continue their new business with; they would have to go back to their old manual methods and loose a lot of face. We would probably help them get the data into a usable state and part on the best possible terms. But this is a moot point anyway; they are very happy and plan a load of future business with us. The point of this thread wasn't to discuss if we had a professional/moral responsibility to provide free redundancy the client isn't willing to pay for, it was to find out if there was any way to improve the update process for a few hours work to make the best of the current servers/software/business processes.

Our company isn't perfect, none is, but we have very strict contracts and change procedures in order to protect us and the clients.

JonRohan, the auditor can come and do whatever he likes, they aren't our servers, we supplied and host them but they belong to the client. I think we are more likely to get in trouble for providing redundancy the client doesn't want because we would be using their data in a way we don't have permission to. Saying we would be to blame is like saying a web hosting company would be to blame if their bottom level server package ever went down. If we had said we will provide x% uptime and then failed to take the necessary precautions to guarantee this we would be to blame but all we have agreed to do is connect their server to the internet and install software updates on it at their request. DPA, backup and security is their problem. We can, and have, given them the benefit of our experience but they have chosen (and stated in a contract) to ignore it.

No matter what angle you look at it from, we are not answerable to audit because we aren't dealing with consumers money, we are dealing with a finance companies money - the obligation lies with them to protect their customers. It would be different if we were managing debt or stock or whatever, but all we are doing is keeping track of money transfers two third parties are making. If we are at fault so are Microsoft for not making excel infallible and me losing 50p from the tuck shop when my machine crashed.

It is interesting that you assume that we are at fault from what little I have told you - We are a big company dealing with big clients dealing with big money, don't you think we are at least a little clued up? Sure, there are rogue companies but they are few and far between at this level of business. Ok, you think redundancy is next to godliness and, I would probably tend to agree, but it's not our place to fork out 20k to provide redundancy to someone who doesn't want it, and it's not your place, nor mine, to condemn the company I work for for not forcing redundancy on our clients against their will.

I would be interested to hear at what point you draw the line of what is our obligation and what is theirs. Do we force them to have 20 char, symbol passwords that are changed on a weekly basis? Do we force them to get 5 forms of identity before accepting a broker onto the system? Do we force them to have every deal independently validated and verified? I think our responsibility goes as far as giving them the best advice we can and telling them when they are making a mistake, much more an we may as well run their business for them.

This isn't a rant, please feel free to argue the point, I would love to hear what you would do in this situation assuming that your boss doesn’t want to spend their profits on redundant servers, client have already said no and wouldn't take kindly to a bill of 20k landing on their doorstep and that you, your boss and your colleagues have tried your best to change the client's mind.

 

[Reezer]

So you basically manage hardware and software that your client provides?

My angle as that of a different one. As we will always 99% of the time chose, install and maintain hardware. I must admit I still don’t quite understand where you are in the loop but it wasn’t where I thought you were, (ie the IT Consultants that manage everything). That’s irrelevant now.

We have clients worth hundreds of millions who also didn't want to fork out the extra thousands for a redundant network. After lot of meetings they finally agreed to it. My point was that if you don't push these kind of things then "if" something goes wrong then you have a lot of work to do and questions to answer regardless of what contracts you have.

I'm not doubting any of your contractual liabilities, as if not seen them as I’m not experienced in that field. I’m sure on something of that size they are pretty good.

But every situation is different and I did jump the gun a bit. It look’s like a rather nice project and I’m sure your company are making some coin out of it which is mostly all that matters.

Did you manage to fix your initial problem?

We also write the software but they are almost completely separate - our biggest customer manages their own servers and we send them updates and they worry about all the hardware/upgrade stuff. But for the customer this thread is about we get a software update request, we spec it up, get it contracted and signed, do the software update, install it on our servers for testing, get it signed off as correct and then shortly after we get a request to install version x.y.z.1 on their live server. We maintain a clear distinction between the development of software and the management of it's use - it's actually quite rare that one person (me) would be involved in both parts; they have just had a major upgrade from beta to V1 which changed a fair few bits of server config so the sysadmin thought it would be best for him to watch and document the upgrade process that I understood so that he could do it in future.

We haven't got a solution to the original problem yet no, we still have to cut them off but we tend to watch the activity log and wait until only a handful of users are on. Once we fix the current bugs we'll do some new development to make it a less "omg i've been cut off!" like no error if it is expected and then retry or set up two websites on the same box as a sort of "software redundancy" allowing us to take one off to update letting the client switch to the other in the meanwhile. Either way it will need a user client build so it won't be for a while.