Labels: 2009, Performance, Sysadmin
I helped some folks the other day with an issue that had the potential to be very serious for them; exposure of production data without someone needing to login to the production system. Not good.
For testing purposes they have copies of their production PeopleSoft databases; one for Financials and one for HCM. These copies are refreshed regularly and there are scripts run against them to handle some cleaning/sanitizing of the production data, so that the database can be used for testing.
They have some people that they allow to login to these test environments with any account so that they can confirm the results of basic processes. The accounts don’t have default passwords, so this access was limited to the people that they authorized to do testing, but testers with access to these cloned environments could login as anyone.
One of the testers (we’ll call him Fred) was logged in to the Financials test environment as some other user (we’ll call her Sally) to test some process that would normally be initiated by Sally in the production system.
While doing this testing, Fred wanted to enter his timesheet data so he clicked on a link to go to the production HCM system for time entry. Much to his surprise, he was logged in to HCM as Sally! So Fred goes back to the test Financials environment, logs in as a different user, clicks the link into production HCM and sure enough he is logged in as that user in production HCM. To Fred’s credit, Fred alerted the PeopleSoft support team there instead of snooping around in HCM.
How did that happen?
The problem comes from the fact that the production environments were configured to trust each other for PeopleSoft Single Signon, but the node names and node passwords were not changed as part of the environment cloning logic.
So when Fred signed on to the cloned Financials environment, the PS_TOKEN cookies generated are identical to what the production environment would generate (the details of PS_TOKEN cookie are documented in PeopleBooks, but the node name and node password are the important pieces here).
How could this be prevented?
There are several different ways of preventing this from happening. Let’s take the pragmatic ones first.
1) Change the node password as part of the refresh script.
This is the easiest, most expedient thing to do. It is the absolute minimum way to solve this problem.
Although this solves the security problem, it introduces a headache for security auditors because the app server logs in the production environment will be full of warnings about failed logins from bad node passwords. That means that testers accessing production while logged in to the test environment will be indistinguishable from someone trying to break in by generating their own PS_TOKEN cookies.
It’s not nice to annoy your security auditors by purposely creating log entries that look like break-in attempts, but are actually OK 🙂
2) Change the node name.
Changing the node name when you clone environments will also solve the problem (because then the production environments will no longer trust the cloned nodes since they don’t recognize the new names). This also solves the log problem mentioned in the previous item.
Changing the node name can have an impact on Integration Broker testing, but that just means that your integration test scripts need to be aware of this. Not a huge drawback in my opinion.
3) Use distinct passwords for each account in the test environment.
This is easy enough to do, it’s mainly a question of distributing the passwords to the testers so that they can get in and do their jobs. Depending on your organization, this may or may not be easy.
4) Don’t provide passwords to the testers, but allow them to reset them on demand.
Similar to the previous item, but gets around the distribution issue by making it an after the fact auditing issue. If Fred requests the CEO’s password in the test system, it’s very easy to audit for that and force Fred to explain why he felt that was necessary.
Computer security is typically focused on preventing people from doing things, but in some cases this model of auditing and disciplining after the fact may be OK.
This model reminds me of when we took our dog to a dog trainer a long time ago that recommended leaving a steak or something else on the kitchen counter for the dog to find. This was in conjunction with hiding around the corner so that at the moment the dog put his paws up to grab the steak, you would jump out and impress upon the dog that it wouldn’t be wise to do that again. A real-time security audit if you will 🙂
What are some of the less pragmatic options?
5) Have a new version of PeopleTools that can detect when a database is copied, and automatically scrub key information like node passwords, GUID, etc.
The implementation of this would be platform specific, but do-able. It would complicate things like having a hot database backup server though. I’m sure those sorts of issues could be addressed though.
6) Embed additional information beyond just node name and node password (like database name) in the PS_TOKEN cookie.
One issue that I can think of with this is that older PeopleTools versions would not be able to interoperate with this, but it could be a flag on the node definition as to whether interoperability with earlier versions of PeopleTools is required. If interoperability was required, then the existing PS_TOKEN format would be used, otherwise the newer format would be used.
Why didn’t they know about this?
PeopleSoft Single Signon is documented in PeopleBooks, but there aren’t any great writeups out there on the implications of it when cloning a PeopleSoft environment.
There are lots of writeups out there on “How to Clone a PeopleSoft Environment”, and some discuss the importance of getting the node configuration correct, but they typically come at from the perspective of just making sure the environment works; not the security implications of it.
PeopleSoft Single Signon is well documented in PeopleBooks, but it just doesn’t seem to jump out at people when thinking about cloning. I will give a shout-out here to Brent Martin of ERP Associates. Brent’s writeup on cloning a PeopleSoft database is the only one out there that I could find that mentions this issue at all.
Labels: 2009, Audit, Security, Sysadmin