Labels: 2009, Performance, Sysadmin
I helped some folks the other day with an issue that had the potential to be very serious for them; exposure of production data without someone needing to login to the production system. Not good.
For testing purposes they have copies of their production PeopleSoft databases; one for Financials and one for HCM. These copies are refreshed regularly and there are scripts run against them to handle some cleaning/sanitizing of the production data, so that the database can be used for testing.
They have some people that they allow to login to these test environments with any account so that they can confirm the results of basic processes. The accounts don’t have default passwords, so this access was limited to the people that they authorized to do testing, but testers with access to these cloned environments could login as anyone.
One of the testers (we’ll call him Fred) was logged in to the Financials test environment as some other user (we’ll call her Sally) to test some process that would normally be initiated by Sally in the production system.
While doing this testing, Fred wanted to enter his timesheet data so he clicked on a link to go to the production HCM system for time entry. Much to his surprise, he was logged in to HCM as Sally! So Fred goes back to the test Financials environment, logs in as a different user, clicks the link into production HCM and sure enough he is logged in as that user in production HCM. To Fred’s credit, Fred alerted the PeopleSoft support team there instead of snooping around in HCM.
How did that happen?
The problem comes from the fact that the production environments were configured to trust each other for PeopleSoft Single Signon, but the node names and node passwords were not changed as part of the environment cloning logic.
So when Fred signed on to the cloned Financials environment, the PS_TOKEN cookies generated are identical to what the production environment would generate (the details of PS_TOKEN cookie are documented in PeopleBooks, but the node name and node password are the important pieces here).
How could this be prevented?
There are several different ways of preventing this from happening. Let’s take the pragmatic ones first.
1) Change the node password as part of the refresh script.
This is the easiest, most expedient thing to do. It is the absolute minimum way to solve this problem.
Although this solves the security problem, it introduces a headache for security auditors because the app server logs in the production environment will be full of warnings about failed logins from bad node passwords. That means that testers accessing production while logged in to the test environment will be indistinguishable from someone trying to break in by generating their own PS_TOKEN cookies.
It’s not nice to annoy your security auditors by purposely creating log entries that look like break-in attempts, but are actually OK 🙂
2) Change the node name.
Changing the node name when you clone environments will also solve the problem (because then the production environments will no longer trust the cloned nodes since they don’t recognize the new names). This also solves the log problem mentioned in the previous item.
Changing the node name can have an impact on Integration Broker testing, but that just means that your integration test scripts need to be aware of this. Not a huge drawback in my opinion.
3) Use distinct passwords for each account in the test environment.
This is easy enough to do, it’s mainly a question of distributing the passwords to the testers so that they can get in and do their jobs. Depending on your organization, this may or may not be easy.
4) Don’t provide passwords to the testers, but allow them to reset them on demand.
Similar to the previous item, but gets around the distribution issue by making it an after the fact auditing issue. If Fred requests the CEO’s password in the test system, it’s very easy to audit for that and force Fred to explain why he felt that was necessary.
Computer security is typically focused on preventing people from doing things, but in some cases this model of auditing and disciplining after the fact may be OK.
This model reminds me of when we took our dog to a dog trainer a long time ago that recommended leaving a steak or something else on the kitchen counter for the dog to find. This was in conjunction with hiding around the corner so that at the moment the dog put his paws up to grab the steak, you would jump out and impress upon the dog that it wouldn’t be wise to do that again. A real-time security audit if you will 🙂
What are some of the less pragmatic options?
5) Have a new version of PeopleTools that can detect when a database is copied, and automatically scrub key information like node passwords, GUID, etc.
The implementation of this would be platform specific, but do-able. It would complicate things like having a hot database backup server though. I’m sure those sorts of issues could be addressed though.
6) Embed additional information beyond just node name and node password (like database name) in the PS_TOKEN cookie.
One issue that I can think of with this is that older PeopleTools versions would not be able to interoperate with this, but it could be a flag on the node definition as to whether interoperability with earlier versions of PeopleTools is required. If interoperability was required, then the existing PS_TOKEN format would be used, otherwise the newer format would be used.
Why didn’t they know about this?
PeopleSoft Single Signon is documented in PeopleBooks, but there aren’t any great writeups out there on the implications of it when cloning a PeopleSoft environment.
There are lots of writeups out there on “How to Clone a PeopleSoft Environment”, and some discuss the importance of getting the node configuration correct, but they typically come at from the perspective of just making sure the environment works; not the security implications of it.
PeopleSoft Single Signon is well documented in PeopleBooks, but it just doesn’t seem to jump out at people when thinking about cloning. I will give a shout-out here to Brent Martin of ERP Associates. Brent’s writeup on cloning a PeopleSoft database is the only one out there that I could find that mentions this issue at all.
Labels: 2009, Audit, Security, Sysadmin
When you run an Application Engine program from within Application Designer (as we mentioned in our blog post on Application Engine Development Tips) and turn on the checkbox for sending output to a log file, it always defaults to sending the output to c:temp.
If you don’t have a c:temp directory though, you’ll end up getting the “Out of Available Memory” error message.
This problem doesn’t occur when running Application Engine via the process scheduler because the process scheduler has already arranged the output directories appropriately (so that each process instance’s output can be tied to the instance number).
It should default the output directory to %TEMP% and just expand that at runtime, but until it does, then the best thing to is just create a c:temp directory so that you don’t have to keep changing it all the time.
When workflow was first added to PeopleSoft 5 back in 1995, the mantra was the three Rs: Rules, Roles, and Routings. I’ll bet that Rick Bergquist still has dreams where he is talking about rules, roles, and routings 🙂
Routings are the part that I wanted to highlight here. The two primary mechanisms for routings that PeopleSoft delivered as part of PeopleSoft 5 were worklists and email.
Worklists are great for people that spend a lot of time in PeopleSoft applications and have enough activity being routed to them that they actually check their worklist, but the broader audience does not typically log in to PeopleSoft to check their worklist to see if there is anything waiting for them to do.
It’s sort of a shame that more work didn’t happen on pushing out PeopleSoft worklist entries to whatever the end-user really use as their “stuff I need to do” list. There are some decent APIs that PeopleSoft delivers these days for accessing that data, but I’m not familiar with them being used for generically pushing PeopleSoft worklists into something like the Todo List functionality in Outlook or Lotus Notes.
In the absence of integrating their worklist entries with something else, most people ended up just using email as the primary mechanism of notifying people that they needed to do something like approving a purchase order in PeopleSoft.
Issues with Workflow Email Notifications.
There are some problems with relying on email though.
Please Do Not Reply To This Email.
Historically the emails generated by PeopleSoft came from a system account, instead of being from an actual person. So PeopleSoft customers would make sure that the emails that were sent out all had text in them telling the person “please do not respond to this email”.
Makes sense, except that the whole reason that we’re sending out emails is because of its universality. Getting an email that you can’t respond to is kind of like getting phone calls from the robot dialers that political campaigns use; pretty annoying.
In newer releases of PeopleSoft, this is less of an issue because there are more places where PeopleSoft application developers took advantage of the ability to generate emails themselves in PeopleCode with more control than the core workflow framework provided. In fact, the Financials/Supply Chain development group did enough of this that they wrote an entire workflow framework using PeopleCode application classes.
In PeopleTools 8.48, that code (known as Approval Workflow Engine) was moved to core PeopleTools so that all PeopleSoft applications and customers could use them. So if you’re up on an applications release 9.0 and higher, be sure to take a look at that.
Stop Sending Me So Many Emails!
We’ve talked with a number of PeopleSoft customers over the years that have ended up turning off workflow because of user complaints regarding the sheer volume of email that they receive. In fact, we just had someone searching our blog today for how to turn off notifications in PeopleSoft HCM.
The problem is that each event that happens (purchase order is entered) and the notification is tied directly to that. Immediate notification is important in some scenarios (confirmation of a customer’s order, employee terminations, etc.), but when someone in management gets 12 expense reports, 15 purchase orders, 9 regularly scheduled reports, etc. showing up in their inbox scattered during the day they sometimes get annoyed.
One way of solving this problem is to re-write your notification processes so that the event and the notification are de-coupled. When the event happens, save off the data (could be in the worklist tables or some other tables) and then have a separate process that delivers the notifications separately.
The problem with re-writing these processes is that it’s time-consuming and expensive, which is why a lot of people end up just turning off the notifications altogether.
The 1990s are calling and they want their email formats back
It’s possible to send HTML emails with PeopleTools, but a lot of the delivered workflow does not take advantage of this. On top of that, the lengthy hyperlinks that can be generated for navigating directly to a particular place in PeopleSoft (the navigation, plus the key values for pulling up the data) are not that attractive.
Who got what and when
Generating notifications as worklist entries leaves a rich history of when the notification was created, worked on, and closed out. There are some nice delivered reports that come with PeopleTools that can show this sort of information. Take a look at the PeopleTools delivered Queries that start with WF. There are some Crystal Reports for those as well.
Tracking email notifications doesn’t happen unless it has been specifically coded into the notification process though. Lots of PeopleSoft applications have added this functionality to key processes, but it’s tied to specific processes. There’s no way to figure out what are all of the emails that a particular user is receiving (or see how many notifications are being generated for a particular process).
The topics discussed in this blog entry come directly from conversations that we have had with different PeopleSoft customers (especially from folks that are familiar with the email functionality in our Report Security and Distribution product; see the Report Notification section on the flash demo). Since we’ve seen continued interest in this topic, we went ahead and built something to fix it.
Announcing the Grey Sparling Email Proxy for PeopleSoft
Here is a high-level overview of what the Email Proxy does:
- It intercepts emails that PeopleSoft generates.
- Classifies them according to your rules (who is getting it, what process is it part of, etc.)
- Optionally rewrites the email
- Provides nicer formatting, rewrites generated links, custom signatures, etc.
- Combine multiple emails together (so someone that gets 20 purchase order notifications in a day can just get one summary email).
There are setup pages where you define the rules that you want for classifying the emails as well as the look and feel of the generated emails, as well as pages for being able to view the emails being generated, statistics regarding the emails, etc.
The Email Proxy server is in beta-testing now at a large PeopleSoft customer that services multi businesses with their PeopleSoft applications and should be generally available shortly.
Continuing on in the cool stuff we’ve been working on series, I wanted to post something about a topic that goes way, way back.
Many years ago, Dave Yoder of Rainbird asked me at a DMUG user conference about how to get variable expansion support for file name references when building a project inside Application Designer.
Without having variables that you can expand, you always have to remember to change the name of the file that will be generated each time you build records/views/indexes, etc. It’s really easy to forget and lose old copies. If you care about saving copies for yourself or for audit purposes, then it gets tiresome to always remember.
Here’s some screenshots of what we have put together to address this particular issue. The first two screenshots show the Application Designer Build Settings dialog with the build log file set to c:temp%YEAR%%MONTH%%DAY%PSBUILD_%DBNAME%.log. Here we are just changing the log file, but it works for any of the files that would get generated when doing builds in Application Designer.
The strings that are between percent signs (YEAR, MONTH, etc.) are variables that get expanded out to real values at the appropriate moment.
Here is what it looks like when you actually run the build.
As you can see, the generated log file has been created as c:temp2009126PSBUILD_PTSYS.log. Since today is January 26, 2009 and I was working in a PTSYS database, that’s exactly what we expected
There are also tokens for things like current hour, minute, second so if you like to do lots of builds in a short time frame you might want to take advantage of those as well.
In addition to things like date/time and connectivity info, there are also tokens that are supported by other Grey Sparling products.
For example, when you are using Grey Sparling Version Control for PeopleSoft you can also do things like reference which ticket number that you are currently working on. Here’s the build settings showing the log file setting as c:tempTicket-%TICKET%PSBUILD_%DBNAME%.log
After running the build (note that our version control plugin is active now; it has to be to supply the current ticket number), you’ll see that the build log file was generated as c:tempTicket-4PSBUILD_PTSYS.log. We could have included the date/time tokens as well if we had wanted to.
Now the build scripts, log files, etc. can easily be associated with the work that you are doing, and even automatically attached to the ticketing system if you want. That helps when your developers and DBAs need an audit trail of these sorts of activities. Fun to use and keeps the suits happy!
So if you run into Dave Yoder at a conference, be sure to thank him for coming up with the idea (and asking about it enough that it finally got built!)
Labels: 2009, ApplicationDesigner, Database, Products, VersionControl